NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely
to seed EnergyRouter centers via np.quantile at load time, which alone
accounted for most of the setup cache sidecar's ~2MB size (float32 values
round-tripped through Python floats serialize at full double precision).
Only a handful of quantile levels are ever read back, so collapse the
sample to a fixed 1001-point quantile grid at save time and interpolate
arbitrary levels from it at use time instead — about 100x smaller with
negligible (<0.001) error on the levels that matter. Bumps the cache
format version since old sidecars have no such grid to fall back on.
Lets the vocab maps, event-id split index, and normalizer stats be
warmed once for a dataset (right after `dwarf convert`, or before a
`dwarf hparam-scan` sweep) without needing to also start training.
Extracts the setup-stage logic out of giant/pipeline.py:run_train_job
into a standalone run_setup_stage() (returning a SetupStageResult),
reused by both run_train_job and the new dwarf command's
scripts/warm_setup_cache.py — a behavior-preserving refactor, covered
by the existing test_pipeline.py suite.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars