- Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of
in-progress weights before exiting mid-epoch — it previously broke
out of the epoch loop before reaching the checkpoint-save block,
contradicting its own printed "saving a checkpoint" message and
losing all progress since the last completed epoch. Checkpoint-dict
construction is factored into a shared _build_checkpoint() helper
used by both the mid-epoch and end-of-epoch save paths.
- WGAN LR-schedule steps_per_epoch used the wrong denominator
(n_critic + 1 instead of n_critic), causing the schedule to exhaust
early and LR to floor to 0 before training completed.
- --critic-lr override was silently dropped on WGAN --resume (only the
generator optimizer's LR was made authoritative again after
load_state_dict; optimizer_d's was not).
- WGAN secondary gradient-penalty forced x_hat/grad to zero for
fully-masked rows (n_sec == 0, common in a shower), adding a
constant ~1.0 bias into the batch-mean GP term; such rows are now
excluded from the mean.
- run_train_job warns (never blocks) when --num-workers exceeds ~1/4
of the machine's CPUs, per this repo's shared-portal-machine
etiquette (see CLAUDE.md's Compute environment section).
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Building the pdg/material vocab maps, the process map, and fitting the
Stage-1/Stage-2 normalizers all require scanning the training dataset
before a single epoch runs, which is wasted work whenever the same
data path is reused across runs (hyperparameter sweeps via `dwarf
hparam-scan`, repeated manual training attempts, ...). Persist those
setup-stage outputs to a JSON sidecar next to the input data
(giant/data/setup_cache.py), validated by a file fingerprint plus
fixed dimension constants and a manually-bumped format version before
reuse, with a soft warning (not a hard invalidation) on a git-hash
mismatch alone.
Also derives n_train_steps instantly from cached per-event row counts
instead of accumulating it during the normalizer scan, and always
collects the energy-router reservoir sample while the cache is being
populated (not only when the current run's router is energy-typed) so
a later run enabling --router-type energy never needs to rescan just
to seed expert centers.
New --cache-setup/--no-cache-setup (default on) and
--rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars