Merged in every non-analysis change from the MoE-prototype branch (routing,
training, data pipeline, streaming rollout output), keeping this branch's
lean streaming giant/analysis.py and rebuilding the rollout-vs-truth feature
natively on it instead of resurrecting the old numpy SampleCollection path.
- Add RolloutVsTruth, accepted anywhere Tier 1-3 functions take a predict-parquet
source: decodes a giant rollout file and a held-out truth file into
RAW_TARGET_NAMES space via a polars port of the forward local-frame rotation,
fully streaming (no SampleCollection, no eager materialization).
- Add compute_rollout_vs_truth_observables_pl for Tier 4, reusing
EventObservables (now backed by independent real_table/gen_table to support
unequal rollout/truth event counts) so every existing shower-observable plot
function works unchanged for both one-step and full-rollout comparisons.
- Update analysis/rollout_validation.ipynb to the new API and CLAUDE.md's
architecture description; add test coverage for the new source type.
- Fix a pre-existing return-type mismatch in giant.rollout.rollout() (found by
`ty check`): the on_chunk summary-dict branch didn't match the declared
dict[str, np.ndarray] return type, now expressed as a RolloutSummary TypedDict.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
_Recorder previously accumulated every generated step across all events/
tracks/steps in Python lists, materialised once at the end and written
via a single pq.write_table — memory scaled with n_events * max_steps *
avg_tracks_per_event. rollout() now takes an optional on_chunk callback
that streams each non-empty batch immediately (fixed per-key dtypes via
_RECORD_DTYPES keep every chunk's table schema identical, which
pq.ParquetWriter requires across writes); giant rollout wires this to an
incrementally-written ParquetWriter, mirroring the row-group streaming
giant predict already does on its input side. Without on_chunk, rollout()
keeps its old buffered return for existing callers/tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
decode_secondaries's stick-breaking only guarantees valid secondary slots
sum to <= e_sec, leaving a shortfall that rollout.py silently dumped into
that step's edep. Rescale the valid slots by one common per-row factor
instead, so they sum to exactly e_sec whenever n_sec > 0: this spreads any
shortfall proportionally across all secondaries rather than concentrating
it in whichever slot is last by energy rank (which would let that one
low-energy secondary balloon and distort the shower's topology). Rows
where every valid slot decodes to ~zero fall back to an even split.
n_sec == 0 rows are unchanged (still nothing to carry the budget, so
rollout.py's edep top-up still applies there) — narrowed the related
caveat in load_rollout_vs_truth's docstring to just that case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the loop from single-step prediction into full showers:
- giant/geometry.py + `dwarf build-geometry-oracle`: learn position ->
(material, layer_id) from data (KNN/SVM) to supply the conditioning the
surrogate does not predict; flag detector escape by NN distance.
- giant/rollout.py: breadth-first batched frontier that steps all active
tracks, spawns secondaries as new tracks, and terminates on energy cutoff,
per-track max steps, escape, or natural end. Energy is deposited locally on
every stop except escape (leakage), so showers conserve energy exactly.
- `giant rollout` CLI: seed from real events (argmax pre_E), load checkpoint,
write a world-frame steps parquet + YAML sidecar.
- giant/analysis.py: compute_rollout_observables + plot_rollout_* for
single-sided longitudinal/transverse/total-energy shower profiles;
analysis/export_rollout_observables.py driver.
- scikit-learn added as an optional `geometry` extra (lazy-imported).
- Tests: tests/test_geometry.py, tests/test_rollout.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
larsandClaude Sonnet 5