dae645120353f9d39e17d2ef0637827f5466ea7a
194 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
dae6451203 |
Apply ruff format
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
ca3a2a3462 |
Fix CLI/tooling robustness gaps and dedupe the Conditioning enum
- run_create_manifest gains --force; it previously overwrote an existing manifest (including holdout.manifest, which check_holdout_overlap exists specifically to protect) with no warning or backup on a second run. - _git_user_name only caught OSError, not subprocess.TimeoutExpired (a SubprocessError, not an OSError) — a slow/loaded shared portal machine could crash `dwarf bump-gen`/`bump-schema` instead of degrading to by=None as intended. - `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when the requested count exceeds ~1/4 of the machine's CPUs, matching the same shared-machine etiquette check added to giant train in the previous commit. - The Conditioning enum was independently redefined in both giant/cli.py and scripts/dwarf.py; moved to a single giant.config.Conditioning both now import, removing the drift risk of a third conditioning mode being added to one but not the other. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
ad1b8e7835 |
Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline
- prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
ad0341a9d4 |
Fix training-loop checkpoint/resume and WGAN bugs
- Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of in-progress weights before exiting mid-epoch — it previously broke out of the epoch loop before reaching the checkpoint-save block, contradicting its own printed "saving a checkpoint" message and losing all progress since the last completed epoch. Checkpoint-dict construction is factored into a shared _build_checkpoint() helper used by both the mid-epoch and end-of-epoch save paths. - WGAN LR-schedule steps_per_epoch used the wrong denominator (n_critic + 1 instead of n_critic), causing the schedule to exhaust early and LR to floor to 0 before training completed. - --critic-lr override was silently dropped on WGAN --resume (only the generator optimizer's LR was made authoritative again after load_state_dict; optimizer_d's was not). - WGAN secondary gradient-penalty forced x_hat/grad to zero for fully-masked rows (n_sec == 0, common in a shower), adding a constant ~1.0 bias into the batch-mean GP term; such rows are now excluded from the mean. - run_train_job warns (never blocks) when --num-workers exceeds ~1/4 of the machine's CPUs, per this repo's shared-portal-machine etiquette (see CLAUDE.md's Compute environment section). Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
5b63dfd588 |
Fix conditioning="physical" so it can actually generalize past training vocab
The whole point of conditioning="physical" is generalizing to a species/material outside the training menu, but two independent code paths still hard-required training-vocab membership: - giant/data/transforms.py: build_cond_features unconditionally raised KeyError on an out-of-vocab pdg/material. _vectorized_map_lookup gains a strict=False mode (dummy index instead of raising), used only under conditioning="physical" where ConditionEncoder never reads cond_cat anyway; "embedding" mode is untouched and still raises, since cond_cat IS the conditioning signal there. - giant/rollout.py: the known_pdg termination gate still killed a track on step 1 for any pdg outside pdg_map, regardless of conditioning mode. Now skipped entirely under conditioning="physical". - giant/model/network.py: PdgRouter/ProcessRouter always build their own training-vocab nn.Embedding independent of conditioning, silently reintroducing the same limitation at the routing layer. build_models now raises loudly if conditioning="physical" is paired with either router type, rather than silently building a model that can't generalize the way it claims to. This unblocks the held-out-species/material generalization experiment against the multi-material dataset (see CLAUDE.md roadmap). Each fix has a regression test, including an end-to-end rollout test seeded with a resolvable-but-out-of-vocab PDG code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
74343d3e48 |
Add data-integrity guards against silent NaN/Inf propagation and races
- log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
a4c0443e01 |
Add bigger WGAN config (hidden_dim=512, n_blocks=6)
Scaled-up variant of the already-trained wgan_h128_b4_physical.toml benchmark config, same training recipe (mode/epochs/lr/warmup), increased model capacity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
22fdca7697 |
Merge pull request 'Add opt-in straight-through Gumbel-softmax combine weights to MoE router' (#26) from feat/router-gumbel-softmax into master
Reviewed-on: #26 |
||
|
|
8065df896e |
Scope wandb run config to only-active hyperparameters
CI / Format (ruff format) (push) Successful in 28s
CI / Lint (ruff check) (push) Successful in 29s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 30s
CI / Type check (ty) (push) Successful in 33s
CI / Format (ruff format) (pull_request) Successful in 37s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 37s
CI / Tests (pull_request) Successful in 1m32s
CI / Tests (push) Successful in 1m36s
Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
5eec4c250a |
Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming
CI / Format (ruff format) (push) Successful in 25s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 29s
CI / Type check (ty) (push) Successful in 31s
CI / Format (ruff format) (pull_request) Successful in 37s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 38s
CI / Tests (push) Successful in 1m34s
CI / Tests (pull_request) Successful in 1m30s
Extends default_out_dir_name's non-default-field convention to the router's new gumbel combine-weight flag and its learnable-knob toggles, so gumbel sweep configs (learn_centers on/off, learn_width, learn_temperature) resolve to distinguishable checkpoint directory names instead of colliding on the same r-<type><n> token. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
af2ee7c7ce |
Add gumbel router configs sweeping learnable-knob combinations
CI / Format (ruff format) (push) Successful in 28s
CI / Lint (ruff check) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 26s
CI / Type check (ty) (push) Successful in 29s
CI / Format (ruff format) (pull_request) Successful in 31s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (push) Successful in 1m39s
CI / Tests (pull_request) Successful in 1m37s
Extends the 10-expert EnergyRouter + physical-conditioning benchmark config with the new opt-in gumbel combine weights, isolating the learn_centers/learn_temperature axis: none, centers only, and centers+temperature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
b51eafcfa5 |
Add opt-in straight-through Gumbel-softmax combine weights to MoE router
CI / Lint (ruff check) (push) Successful in 28s
CI / Format (ruff format) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 21s
CI / Lint (ruff check) (pull_request) Successful in 25s
CI / Format (ruff format) (pull_request) Successful in 30s
CI / Tests (push) Successful in 1m37s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (pull_request) Successful in 55s
Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
43cb6dd9ae |
Merge pull request 'Add learnable per-expert width and shared temperature to EnergyRouter' (#25) from feat/router-learnable-width into master
Reviewed-on: #25 |
||
|
|
aa55c407ab |
Merge branch 'master' into feat/router-learnable-width
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 28s
CI / Type check (ty) (push) Successful in 30s
CI / Format (ruff format) (pull_request) Successful in 33s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 32s
CI / Tests (pull_request) Successful in 1m38s
CI / Tests (push) Successful in 1m42s
|
||
|
|
da5f54ea1c |
Add learnable per-expert width and shared temperature to EnergyRouter
CI / Lint (ruff check) (push) Successful in 28s
CI / Format (ruff format) (push) Successful in 29s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 27s
CI / Tests (push) Successful in 1m1s
CI / Lint (ruff check) (pull_request) Successful in 30s
CI / Format (ruff format) (pull_request) Successful in 27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 27s
CI / Tests (pull_request) Successful in 59s
EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
78297456e3 |
Merge pull request 'Store a quantile grid instead of a raw reservoir sample in the setup cache' (#24) from perf/quantile-energy-reservoir into master
Reviewed-on: #24 |
||
|
|
d656cf3109 |
Store a quantile grid instead of a raw reservoir sample in the setup cache
CI / Format (ruff format) (push) Successful in 26s
CI / Lint (ruff check) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 24s
CI / Tests (push) Successful in 58s
CI / Format (ruff format) (pull_request) Successful in 28s
CI / Lint (ruff check) (pull_request) Successful in 28s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 28s
CI / Tests (pull_request) Successful in 59s
NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely to seed EnergyRouter centers via np.quantile at load time, which alone accounted for most of the setup cache sidecar's ~2MB size (float32 values round-tripped through Python floats serialize at full double precision). Only a handful of quantile levels are ever read back, so collapse the sample to a fixed 1001-point quantile grid at save time and interpolate arbitrary levels from it at use time instead — about 100x smaller with negligible (<0.001) error on the levels that matter. Bumps the cache format version since old sidecars have no such grid to fall back on. |
||
|
|
de5db25e3f |
Add giant new-run to scaffold a config.toml + run dir ahead of training
Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch: `new-run` resolves CLI hyperparameter overrides into a full config.toml and run dir (reusing the existing default_out_dir_name collision-avoidance and a newly factored-out router-override helper shared with `train`), so a run can be prepared and reviewed before `giant train` actually kicks off. Also brings README up to date with the model/CLI as it actually stands (physical/embedding conditioning, WGAN/MoE-router modes, giant analyze, W&B, setup-stage caching), which had drifted back to describing the Phase-1 proof-of-concept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
1fd2625889 |
Merge pull request 'Offset event_id per file to avoid cross-file collisions' (#23) from fix/event-id-file-offset into master
Reviewed-on: #23 |
||
|
|
b944bba8fb |
Offset event_id per file to avoid cross-file collisions
CI / Format (ruff format) (push) Successful in 25s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 23s
CI / Lint (ruff check) (pull_request) Successful in 27s
CI / Tests (push) Successful in 1m14s
CI / Format (ruff format) (pull_request) Successful in 27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (pull_request) Successful in 59s
Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
e288c3fe21 |
Merge pull request 'Cache giant train's setup stage in a sidecar file' (#21) from perf/setup-stage into master
Reviewed-on: #21 |
||
|
|
5aaf6cde4d |
Merge branch 'master' into perf/setup-stage
CI / Format (ruff format) (push) Successful in 25s
CI / Lint (ruff check) (push) Successful in 25s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 31s
CI / Type check (ty) (push) Successful in 40s
CI / Format (ruff format) (pull_request) Successful in 38s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 33s
CI / Tests (pull_request) Successful in 1m28s
CI / Tests (push) Successful in 1m34s
|
||
|
|
471a81b5e7 |
Add dwarf warm-cache to precompute the setup-stage sidecar
CI / Format (ruff format) (push) Successful in 26s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 37s
CI / Lint (ruff check) (pull_request) Successful in 36s
CI / Format (ruff format) (pull_request) Successful in 31s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 33s
CI / Tests (push) Successful in 1m43s
CI / Tests (pull_request) Successful in 1m39s
Lets the vocab maps, event-id split index, and normalizer stats be warmed once for a dataset (right after `dwarf convert`, or before a `dwarf hparam-scan` sweep) without needing to also start training. Extracts the setup-stage logic out of giant/pipeline.py:run_train_job into a standalone run_setup_stage() (returning a SetupStageResult), reused by both run_train_job and the new dwarf command's scripts/warm_setup_cache.py — a behavior-preserving refactor, covered by the existing test_pipeline.py suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
26aa9d3fde |
Pass --seed through to the train/val event split
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 28s
CI / Type check (ty) (push) Successful in 39s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 41s
CI / Format (ruff format) (pull_request) Successful in 39s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 37s
CI / Tests (push) Successful in 1m56s
CI / Tests (pull_request) Successful in 1m27s
make_event_split() defaults to seed=42, and run_train_job was calling it without forwarding t["seed"] — so the configured --seed affected model init/EMA/etc. but not which events landed in train vs. val, which silently always used seed 42 regardless of --seed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
e7478c36fb |
Cache giant train's setup stage in a sidecar file
Building the pdg/material vocab maps, the process map, and fitting the Stage-1/Stage-2 normalizers all require scanning the training dataset before a single epoch runs, which is wasted work whenever the same data path is reused across runs (hyperparameter sweeps via `dwarf hparam-scan`, repeated manual training attempts, ...). Persist those setup-stage outputs to a JSON sidecar next to the input data (giant/data/setup_cache.py), validated by a file fingerprint plus fixed dimension constants and a manually-bumped format version before reuse, with a soft warning (not a hard invalidation) on a git-hash mismatch alone. Also derives n_train_steps instantly from cached per-event row counts instead of accumulating it during the normalizer scan, and always collects the energy-router reservoir sample while the cache is being populated (not only when the current run's router is energy-typed) so a later run enabling --router-type energy never needs to rescan just to seed expert centers. New --cache-setup/--no-cache-setup (default on) and --rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
9112625a08 |
Merge pull request 'Speed up giant train's setup stage' (#20) from perf/setup-stage into master
Reviewed-on: #20 |
||
|
|
84efbf5c2c |
Merge branch 'master' into perf/setup-stage
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 25s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 32s
CI / Type check (ty) (push) Successful in 34s
CI / Format (ruff format) (pull_request) Successful in 41s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 41s
CI / Tests (push) Successful in 2m1s
CI / Tests (pull_request) Successful in 1m57s
|
||
|
|
759b67a9e1 |
Speed up _WelfordAccumulator's per-chunk update
CI / Format (ruff format) (push) Successful in 27s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 35s
CI / Type check (ty) (push) Successful in 38s
CI / Format (ruff format) (pull_request) Successful in 33s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 33s
CI / Tests (pull_request) Successful in 1m21s
CI / Tests (push) Successful in 1m27s
The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
47a6c9db1f |
Add regression coverage for vocab/process index-map builders
CI / Format (ruff format) (push) Successful in 26s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 26s
CI / Type check (ty) (push) Successful in 29s
CI / Format (ruff format) (pull_request) Successful in 36s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 36s
CI / Tests (push) Successful in 1m44s
CI / Tests (pull_request) Successful in 1m41s
build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
e331148afa |
Merge pull request 'Make default checkpoint out_dir name reflect only non-default hyperparams' (#19) from feature/smart-outdir-naming into master
Reviewed-on: #19 |
||
|
|
09e4c765c7 |
Speed up giant train's setup stage
CI / Format (ruff format) (push) Successful in 26s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 27s
CI / Tests (push) Successful in 51s
CI / Lint (ruff check) (pull_request) Successful in 28s
CI / Format (ruff format) (pull_request) Successful in 26s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 33s
CI / Tests (pull_request) Successful in 57s
Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
1115451c8e |
Make default checkpoint out_dir name reflect only non-default hyperparams
CI / Format (ruff format) (push) Successful in 27s
CI / Lint (ruff check) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 23s
CI / Tests (push) Successful in 53s
CI / Format (ruff format) (pull_request) Successful in 28s
CI / Lint (ruff check) (pull_request) Successful in 28s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 22s
CI / Tests (pull_request) Successful in 52s
Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs. |
||
|
|
f2f89023d5 |
Merge pull request 'Feature/wandb integration' (#18) from feature/wandb-integration into master
Reviewed-on: #18 |
||
|
|
4c19072724 |
Skip empty-slice mean/std in sec phys validation print
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 35s
CI / Type check (ty) (push) Successful in 37s
CI / Format (ruff format) (pull_request) Successful in 36s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 36s
CI / Tests (pull_request) Successful in 1m17s
CI / Tests (push) Successful in 1m23s
The per-dim print loop lacked the empty-array guard already used for the KL computation right above it and the sec-slot loop further down, so an all-zero-secondaries validation batch (e.g. early/unstable training) triggered numpy RuntimeWarnings from .mean()/.std() on empty arrays. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
539b6f61e1 |
Add test coverage for resolve_expert_dims
CI / Lint (ruff check) (push) Successful in 28s
CI / Format (ruff format) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 40s
CI / Type check (ty) (push) Successful in 43s
CI / Format (ruff format) (pull_request) Successful in 39s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 41s
CI / Tests (push) Successful in 1m26s
CI / Tests (pull_request) Successful in 1m18s
Covers the default-config 0-sentinel inheritance path (the exact bug
fixed by
|
||
|
|
f427d3384f |
Timestamp default checkpoint dir to avoid W&B run-id collisions
out_dir (and thus the W&B run id, which is derived from out_dir.name) was previously date-only, so two fresh runs on the same day with identical hyperparams silently shared one W&B run history. Default out_dir is now timestamped to the second. --resume without an explicit --out now reuses the checkpoint's own parent directory instead of recomputing a hyperparam-derived name, which both preserves the old continue-in-place behavior and fixes a latent bug where a resumed run with a changed hyperparam (e.g. --lr) would silently start writing to a new directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
bb699d41b2 |
Persist global_step across --resume so W&B step stays monotonic
Previously global_step always reset to 0, even on --resume. Since the W&B run reattaches to the same run id on resume, logging with step=global_step after a restart passed step values below what was already recorded, silently dropping the resumed portion's metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
a986f96ba3 |
Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B
CI / Lint (ruff check) (push) Successful in 35s
CI / Format (ruff format) (push) Successful in 35s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 26s
CI / Type check (ty) (push) Successful in 27s
CI / Format (ruff format) (pull_request) Successful in 38s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 38s
CI / Tests (push) Successful in 1m26s
CI / Tests (pull_request) Successful in 1m24s
Adds Router.gate_stats (per-router gate entropy + per-expert utilization), logged both per-batch (entropy only, train loop) and per-epoch (full stats, over the whole val set) — the router-collapse failure mode from the roadmap's rollout postmortem is now visible during training instead of only after a full rollout+analysis run. Also splits WGAN critic/ generator grad norms instead of summing them, logs critic LR, n_sec head accuracy, GPU peak memory + samples/sec, model parameter counts (in wandb.config), and an is_best flag — all wired into both metrics.csv and W&B. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
969c5c6e9a |
Fix router experts silently ignoring --hidden-dim/--n-blocks
CI / Format (ruff format) (push) Successful in 27s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 31s
CI / Type check (ty) (push) Successful in 34s
CI / Format (ruff format) (pull_request) Successful in 41s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 41s
CI / Tests (push) Successful in 1m36s
CI / Tests (pull_request) Successful in 1m33s
expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in
DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed
run always got fixed 128/3-wide experts no matter what --hidden-dim/
--n-blocks was passed. They now default to 0 ("unset"), which
resolve_expert_dims() resolves by inheriting the model dims; an
explicit override still works and now warns when it diverges from
model.hidden_dim/n_blocks, since the checkpoint dir name won't
reflect it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
|
29459ab1f7 |
Log batch-level metrics to W&B, not just per-epoch summaries
CI / Format (ruff format) (push) Successful in 26s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 25s
CI / Type check (ty) (push) Successful in 28s
CI / Format (ruff format) (pull_request) Successful in 31s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (push) Successful in 1m33s
CI / Tests (pull_request) Successful in 1m29s
giant train --wandb now also logs loss/grad_norm/lr every N optimizer steps (--wandb-log-every, default 50) so W&B shows within-epoch trends, not just one point per epoch. Both share global_step as a single monotonic step axis (wandb.Run.log requires step to never decrease across calls), which also fixes global_step previously only advancing in wgan mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
a05837f918 |
Apply ruff format
CI / Lint (ruff check) (push) Successful in 25s
CI / Format (ruff format) (push) Successful in 26s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 23s
CI / Tests (push) Successful in 53s
CI / Lint (ruff check) (pull_request) Successful in 24s
CI / Format (ruff format) (pull_request) Successful in 28s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 23s
CI / Tests (pull_request) Successful in 56s
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
0778a61360 |
Add opt-in Weights & Biases logging for the training loop
giant train --wandb logs the same per-epoch metrics already written to metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`) that nothing else depends on. A run's id is derived from the checkpoint out_dir so --resume reattaches to the existing run instead of starting a new one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
db0f12be58 |
Add configs for router energy (embedding/physical) and WGAN baseline runs
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
6208810089 |
Merge pull request 'Refactor the analysis plot creation with focus on rollout' (#16) from analysis-rollout-plots into master
Reviewed-on: #16 |
||
|
|
5594866eac |
Merge remote-tracking branch 'origin/master' into analysis-rollout-plots
CI / Lint (ruff check) (push) Successful in 26s
CI / Format (ruff format) (push) Successful in 26s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 28s
CI / Type check (ty) (push) Successful in 31s
CI / Format (ruff format) (pull_request) Successful in 36s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 36s
CI / Tests (push) Successful in 1m23s
CI / Tests (pull_request) Successful in 1m22s
# Conflicts: # analysis/rollout_validation.ipynb |
||
|
|
0f7febfd02 |
ci: set UV_LINK_MODE=copy to silence the cross-filesystem hardlink warning
CI / Lint (ruff check) (push) Successful in 33s
CI / Format (ruff format) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 41s
CI / Lint (ruff check) (pull_request) Successful in 38s
CI / Format (ruff format) (pull_request) Successful in 33s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 34s
CI / Tests (push) Successful in 2m5s
CI / Tests (pull_request) Successful in 2m4s
The bind-mounted uv cache and the job workspace are on different filesystems, so uv already falls back to copying installed files; this just tells it to do so directly instead of logging a hardlink-failed warning every job. |
||
|
|
19be455346 |
ci: re-pin UV_CACHE_DIR after setup-uv, which exports its own value regardless of enable-cache
CI / Format (ruff format) (push) Successful in 1m34s
CI / Lint (ruff check) (push) Successful in 1m43s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 45s
CI / Lint (ruff check) (pull_request) Successful in 42s
CI / Format (ruff format) (pull_request) Successful in 45s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 47s
CI / Tests (push) Successful in 1m39s
CI / Tests (pull_request) Successful in 1m46s
Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path (/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step right after setup-uv so it wins for the rest of the job. |
||
|
|
1787d26d1c |
ci: stop setup-uv from overriding UV_CACHE_DIR
CI / Lint (ruff check) (push) Successful in 1m53s
CI / Format (ruff format) (push) Successful in 1m30s
CI / Type check (ty) (push) Successful in 1m25s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 1m12s
CI / Type check (ty) (pull_request) Successful in 1m27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Format (ruff format) (pull_request) Successful in 1m59s
CI / Tests (push) Successful in 3m38s
CI / Tests (pull_request) Successful in 2m22s
setup-uv's default enable-cache: auto sets its own UV_CACHE_DIR (a tool-cache tmp path) as a later step, clobbering the workflow-level UV_CACHE_DIR that points at the bind-mounted persistent cache. Disable setup-uv's own cache handling so our mount stays in effect. |
||
|
|
0afa75ee30 |
ci: replace unsupported artifact sharing with a bind-mounted uv cache
CI / Format (ruff format) (push) Successful in 1m43s
CI / Lint (ruff check) (push) Successful in 1m43s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 1m53s
CI / Type check (ty) (push) Successful in 1m58s
CI / Format (ruff format) (pull_request) Successful in 1m47s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 1m52s
CI / Tests (push) Successful in 5m40s
CI / Tests (pull_request) Successful in 3m48s
The self-hosted act_runner doesn't support upload/download-artifact, so drop that plumbing and instead run jobs in an explicit container with a persistent host directory mounted at /uv-cache (UV_CACHE_DIR), backed by valid_volumes on the runner. uv sync still runs per job but hits a warm local cache instead of re-downloading/building packages every time. |
||
|
|
08c76a9614 |
ci: share one uv sync across jobs, gate tests on lint+type-check, sync tag/version on release tags
CI / Setup environment (push) Failing after 2m28s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (push) Has been skipped
CI / Format (ruff format) (push) Has been skipped
CI / Type check (ty) (push) Has been skipped
CI / Tests (push) Has been skipped
CI / Setup environment (pull_request) Failing after 2m27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Lint (ruff check) (pull_request) Has been skipped
CI / Format (ruff format) (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Has been skipped
CI / Tests (pull_request) Has been skipped
Uploads the synced .venv as an artifact from a single setup job instead of re-running uv sync (and re-downloading torch) in every job. Drops the build/publish job in favor of a lighter job that, on a pushed tag, checks the tag against the uv project version and — if they differ — bumps the version, commits it to master, and recreates the tag on the new commit. |