Building the pdg/material vocab maps, the process map, and fitting the
Stage-1/Stage-2 normalizers all require scanning the training dataset
before a single epoch runs, which is wasted work whenever the same
data path is reused across runs (hyperparameter sweeps via `dwarf
hparam-scan`, repeated manual training attempts, ...). Persist those
setup-stage outputs to a JSON sidecar next to the input data
(giant/data/setup_cache.py), validated by a file fingerprint plus
fixed dimension constants and a manually-bumped format version before
reuse, with a soft warning (not a hard invalidation) on a git-hash
mismatch alone.
Also derives n_train_steps instantly from cached per-event row counts
instead of accumulating it during the normalizer scan, and always
collects the energy-router reservoir sample while the cache is being
populated (not only when the current run's router is energy-typed) so
a later run enabling --router-type energy never needs to rescan just
to seed expert centers.
New --cache-setup/--no-cache-setup (default on) and
--rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars
e7478c36fb