The venv-detection change in condor.py now checks for a giant binary next
to sys.executable before falling back to repo_dir/.venv/bin/giant, and the
test's own pytest venv has one — so the expected FileNotFoundError never
fired. Monkeypatch sys.executable to a nonexistent path so the test exercises
the fallback with neither location populated.
write_submit baked in cfg.repo_dir/.venv/bin/giant unconditionally, which
breaks when submitting from a differently-named or non-default venv (e.g.
--extra cuda). Prefer the giant executable next to sys.executable (the venv
actually running the submit), falling back to repo_dir/.venv/bin/giant.
- run_create_manifest gains --force; it previously overwrote an
existing manifest (including holdout.manifest, which
check_holdout_overlap exists specifically to protect) with no
warning or backup on a second run.
- _git_user_name only caught OSError, not subprocess.TimeoutExpired (a
SubprocessError, not an OSError) — a slow/loaded shared portal
machine could crash `dwarf bump-gen`/`bump-schema` instead of
degrading to by=None as intended.
- `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when
the requested count exceeds ~1/4 of the machine's CPUs, matching the
same shared-machine etiquette check added to giant train in the
previous commit.
- The Conditioning enum was independently redefined in both
giant/cli.py and scripts/dwarf.py; moved to a single
giant.config.Conditioning both now import, removing the drift risk
of a third conditioning mode being added to one but not the other.
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- prep() now clears reduced_partial/ and reduced/ on every (re-)run.
Partial files carry no record of what context (n_chunks, bin edges,
group sets) they were computed under, so re-prepping the same run_dir
with a different --chunks/--bins/--top-pdg (or after the rollout was
regenerated) previously left old partials on disk that merge_one
would silently merge against the new shared.json — producing a
wrong-but-plausible reduced/*.json with no error.
- write_submit() now checks SubmitConfig.n_chunks against the run
directory's own RunMeta.n_chunks (fixed at prep time, and what
rows_per_chunk is sized against) and raises a clear error on
mismatch, instead of an uncaught IndexError deep in _job_walltimes.
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of
in-progress weights before exiting mid-epoch — it previously broke
out of the epoch loop before reaching the checkpoint-save block,
contradicting its own printed "saving a checkpoint" message and
losing all progress since the last completed epoch. Checkpoint-dict
construction is factored into a shared _build_checkpoint() helper
used by both the mid-epoch and end-of-epoch save paths.
- WGAN LR-schedule steps_per_epoch used the wrong denominator
(n_critic + 1 instead of n_critic), causing the schedule to exhaust
early and LR to floor to 0 before training completed.
- --critic-lr override was silently dropped on WGAN --resume (only the
generator optimizer's LR was made authoritative again after
load_state_dict; optimizer_d's was not).
- WGAN secondary gradient-penalty forced x_hat/grad to zero for
fully-masked rows (n_sec == 0, common in a shower), adding a
constant ~1.0 bias into the batch-mean GP term; such rows are now
excluded from the mean.
- run_train_job warns (never blocks) when --num-workers exceeds ~1/4
of the machine's CPUs, per this repo's shared-portal-machine
etiquette (see CLAUDE.md's Compute environment section).
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The whole point of conditioning="physical" is generalizing to a
species/material outside the training menu, but two independent code
paths still hard-required training-vocab membership:
- giant/data/transforms.py: build_cond_features unconditionally raised
KeyError on an out-of-vocab pdg/material. _vectorized_map_lookup
gains a strict=False mode (dummy index instead of raising), used only
under conditioning="physical" where ConditionEncoder never reads
cond_cat anyway; "embedding" mode is untouched and still raises,
since cond_cat IS the conditioning signal there.
- giant/rollout.py: the known_pdg termination gate still killed a track
on step 1 for any pdg outside pdg_map, regardless of conditioning
mode. Now skipped entirely under conditioning="physical".
- giant/model/network.py: PdgRouter/ProcessRouter always build their
own training-vocab nn.Embedding independent of conditioning, silently
reintroducing the same limitation at the routing layer. build_models
now raises loudly if conditioning="physical" is paired with either
router type, rather than silently building a model that can't
generalize the way it claims to.
This unblocks the held-out-species/material generalization experiment
against the multi-material dataset (see CLAUDE.md roadmap). Each fix
has a regression test, including an end-to-end rollout test seeded
with a resolvable-but-out-of-vocab PDG code.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- log_transform / _validate_unit_pre_dir now raise on non-finite input
instead of letting a NaN row silently poison the persisted normalizer
cache (norm < 1e-6 was always False for NaN, so the existing guard
never caught it).
- encode_secondaries warns when a row's secondary energies cumulatively
exceed e_sec, instead of silently saturating the overflowing slot's
stick-breaking logit via the _EPS floor.
- EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding
two files' event ids together (reintroducing train/val leakage).
- make_event_split(val_fraction=0.0) now actually holds out nothing,
instead of always forcing at least 1 validation event.
- setup_cache.save() is now serialized with a flock, since two
concurrent writers (a real scenario on this repo's shared
portal/condor machines) could otherwise race and silently drop one
writer's freshly-computed cache section.
- Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation
singularity in _rodrigues_axis, which is real but inherent to any
single-valued local-frame convention.
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Scaled-up variant of the already-trained wgan_h128_b4_physical.toml
benchmark config, same training recipe (mode/epochs/lr/warmup),
increased model capacity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/
gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were
being logged to wandb's top-level run config unconditionally, even for
runs where routing or WGAN mode is off, implying hyperparameters from
an inactive code path. Extract _wandb_run_config and only include each
group when its gate is actually true (router.enabled / mode=="wgan");
the full model_config (with its router sub-dict) is still always
logged in full, so no information is lost.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends default_out_dir_name's non-default-field convention to the
router's new gumbel combine-weight flag and its learnable-knob toggles,
so gumbel sweep configs (learn_centers on/off, learn_width,
learn_temperature) resolve to distinguishable checkpoint directory
names instead of colliding on the same r-<type><n> token.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the 10-expert EnergyRouter + physical-conditioning benchmark
config with the new opt-in gumbel combine weights, isolating the
learn_centers/learn_temperature axis: none, centers only, and
centers+temperature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Trains the routed trunk's forward combination as a hard one-hot sample
(matching eval-time top-1 dispatch exactly) while keeping a smooth gradient
on the backward pass, targeting the train/eval mismatch identified as a
likely contributor to experts overlapping instead of partitioning in the
first energy-router rollout benchmark. Off by default (model.router.gumbel);
existing routed configs/checkpoints are unaffected.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
EnergyRouter's gate sharpness was a single fixed temperature shared by
every expert, with no way for an expert to independently learn how much
of the energy axis it covers. Adds two mutually exclusive, default-off
modes: learn_width (per-expert learnable width) and learn_temperature
(single learnable shared scalar), both bounded via a sigmoid
interpolation warm-started to reproduce today's fixed-temperature gate
exactly at init, to compare against each other without risking the
unbounded-width collapse failure mode. Also promotes gate_stats's
entropy into a generic, optional Router.entropy_loss (lambda_entropy) as
a secondary guard against all experts' widths co-inflating together.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely
to seed EnergyRouter centers via np.quantile at load time, which alone
accounted for most of the setup cache sidecar's ~2MB size (float32 values
round-tripped through Python floats serialize at full double precision).
Only a handful of quantile levels are ever read back, so collapse the
sample to a fixed 1001-point quantile grid at save time and interpolate
arbitrary levels from it at use time instead — about 100x smaller with
negligible (<0.001) error on the levels that matter. Bumps the cache
format version since old sidecars have no such grid to fall back on.
Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch:
`new-run` resolves CLI hyperparameter overrides into a full config.toml and
run dir (reusing the existing default_out_dir_name collision-avoidance and a
newly factored-out router-override helper shared with `train`), so a run can
be prepared and reviewed before `giant train` actually kicks off. Also
brings README up to date with the model/CLI as it actually stands
(physical/embedding conditioning, WGAN/MoE-router modes, giant analyze,
W&B, setup-stage caching), which had drifted back to describing the
Phase-1 proof-of-concept.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py),
and a job's event_id numbering always restarts from 0 — so loading
multiple files together (a directory or .manifest) let same-numbered
events from different files collapse into one during the event index
scan and train/val split, corrupting both. Every per-file event_id now
gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py),
threaded through the setup-cache event index, the streaming dataset,
and predict/rollout seeding. Bumps the setup-cache format version so
stale sidecars computed pre-fix are invalidated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Lets the vocab maps, event-id split index, and normalizer stats be
warmed once for a dataset (right after `dwarf convert`, or before a
`dwarf hparam-scan` sweep) without needing to also start training.
Extracts the setup-stage logic out of giant/pipeline.py:run_train_job
into a standalone run_setup_stage() (returning a SetupStageResult),
reused by both run_train_job and the new dwarf command's
scripts/warm_setup_cache.py — a behavior-preserving refactor, covered
by the existing test_pipeline.py suite.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
make_event_split() defaults to seed=42, and run_train_job was calling
it without forwarding t["seed"] — so the configured --seed affected
model init/EMA/etc. but not which events landed in train vs. val,
which silently always used seed 42 regardless of --seed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Building the pdg/material vocab maps, the process map, and fitting the
Stage-1/Stage-2 normalizers all require scanning the training dataset
before a single epoch runs, which is wasted work whenever the same
data path is reused across runs (hyperparameter sweeps via `dwarf
hparam-scan`, repeated manual training attempts, ...). Persist those
setup-stage outputs to a JSON sidecar next to the input data
(giant/data/setup_cache.py), validated by a file fingerprint plus
fixed dimension constants and a manually-bumped format version before
reuse, with a soft warning (not a hard invalidation) on a git-hash
mismatch alone.
Also derives n_train_steps instantly from cached per-event row counts
instead of accumulating it during the normalizer scan, and always
collects the energy-router reservoir sample while the cache is being
populated (not only when the current run's router is energy-typed) so
a later run enabling --router-type energy never needs to rescan just
to seed expert centers.
New --cache-setup/--no-cache-setup (default on) and
--rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The streaming update re-derived two full (B, F) arrays from the
running mean (once before updating it, once after) plus an elementwise
product — five passes over each chunk and three temporary arrays, to
maintain a mean/variance that's tiny in width (COND_DIM=15 at most).
Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm:
compute the chunk's own local mean/M2 (independent of the running
state) and merge it in with an O(F) combination formula. Same
streaming interface and output (identical to ~1e-14, float64 rounding
noise), ~40% faster per update() call on a benchmark chunk.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build_index_maps, build_index_maps_from_files, and (mostly)
build_process_map_from_files had no test pinning their sort order,
tie-breaking, or cross-file union behavior — all load-bearing for a
trained checkpoint's vocabulary, and all at risk of silently changing
under a future single-pass (pyarrow/polars) rewrite of the setup-stage
scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion
codes), negative PDG codes, dedup/bijective indices, file-order
independence, and process-map tie-breaking/boundary conditions
(n_experts=1, fewer processes than experts, 3-file partial overlap).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fits normalizers over multi-hundred-million-row datasets, so the setup
pass's per-row Python overhead compounds fast: encode_secondaries
recomputed an O(K) prefix sum from scratch on every one of its 15
stick-breaking iterations, np.isin re-sorted the full train-event-id
array on every chunk, and pdg/material/process index lookups ran a
Python dict lookup per row. The normalizer-fit pass also computed
encode_secondaries's stick-logit and direction-rotation blocks in full
even though it only ever reads the mass/charge columns.
Replace the prefix-sum recompute with a single np.cumsum, add a
sorted_membership helper (searchsorted-based) in place of np.isin at
both the setup-pass and per-epoch call sites, vectorize the index
lookups via _vectorized_map_lookup, and add an opt-in phys_only path
so the setup pass skips the stick-breaking/rotation work it discards
anyway. All four changes are output-identical performance refactors,
backed by new unit tests plus the existing suite.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/
conditioning/lr/batch_size) were always baked into the name, even for a
vanilla run, and router config wasn't represented at all. Now
default_out_dir_name only includes fields that differ from
DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6
shown fields with a hashed overflow suffix for heavily-swept configs.
The per-dim print loop lacked the empty-array guard already used for
the KL computation right above it and the sec-slot loop further down,
so an all-zero-secondaries validation batch (e.g. early/unstable
training) triggered numpy RuntimeWarnings from .mean()/.std() on
empty arrays.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the default-config 0-sentinel inheritance path (the exact bug
fixed by 969c5c6, previously untested since every router test in
test_router.py passes expert_hidden_dim/expert_n_blocks explicitly),
plus missing-key inheritance, full override, and partial override.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
out_dir (and thus the W&B run id, which is derived from out_dir.name)
was previously date-only, so two fresh runs on the same day with
identical hyperparams silently shared one W&B run history. Default
out_dir is now timestamped to the second. --resume without an explicit
--out now reuses the checkpoint's own parent directory instead of
recomputing a hyperparam-derived name, which both preserves the old
continue-in-place behavior and fixes a latent bug where a resumed run
with a changed hyperparam (e.g. --lr) would silently start writing to
a new directory.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously global_step always reset to 0, even on --resume. Since the
W&B run reattaches to the same run id on resume, logging with
step=global_step after a restart passed step values below what was
already recorded, silently dropping the resumed portion's metrics.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds Router.gate_stats (per-router gate entropy + per-expert utilization),
logged both per-batch (entropy only, train loop) and per-epoch (full
stats, over the whole val set) — the router-collapse failure mode from
the roadmap's rollout postmortem is now visible during training instead
of only after a full rollout+analysis run. Also splits WGAN critic/
generator grad norms instead of summing them, logs critic LR, n_sec head
accuracy, GPU peak memory + samples/sec, model parameter counts (in
wandb.config), and an is_best flag — all wired into both metrics.csv and
W&B.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in
DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed
run always got fixed 128/3-wide experts no matter what --hidden-dim/
--n-blocks was passed. They now default to 0 ("unset"), which
resolve_expert_dims() resolves by inheriting the model dims; an
explicit override still works and now warns when it diverges from
model.hidden_dim/n_blocks, since the checkpoint dir name won't
reflect it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant train --wandb now also logs loss/grad_norm/lr every N optimizer
steps (--wandb-log-every, default 50) so W&B shows within-epoch trends,
not just one point per epoch. Both share global_step as a single
monotonic step axis (wandb.Run.log requires step to never decrease
across calls), which also fixes global_step previously only advancing
in wgan mode.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant train --wandb logs the same per-epoch metrics already written to
metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`)
that nothing else depends on. A run's id is derived from the checkpoint
out_dir so --resume reattaches to the existing run instead of starting
a new one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The bind-mounted uv cache and the job workspace are on different
filesystems, so uv already falls back to copying installed files; this just
tells it to do so directly instead of logging a hardlink-failed warning
every job.
Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path
(/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow
env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step
right after setup-uv so it wins for the rest of the job.
lars
