Add a second parallelism axis to giant analyze: each plot's data can now
be split into a configurable number of event_id-disjoint chunks, each
computed as its own HTCondor job, bounding per-job walltime and scan cost
on large rollout/reference files instead of one job re-scanning the
whole file per plot.
Every PlotSpec now splits into compute_partial (runs per (plot, chunk)
job against a chunk-filtered Bundle) and finalize (merges chunks -
elementwise sum for fixed-edge histograms/species shares, concatenate
-then-recompute for specs that derive edges or mean/std from the full
per-event/per-secondary array). Router diagnostics stay chunkable=False
and always run as a single job. giant analyze render now joins every
plot's chunk partials (merge_all) before rendering, transparently.
New: --chunks on `analyze prep`/`analyze submit`, --chunk on
`analyze compute-one`, and a new `analyze merge-one` command.
New "model" family in the gallery: router_gating (mean soft gate weight
vs. pre-step energy, showing the router's soft decision boundaries) and
router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share
by species / true physics process). Needs a live checkpoint's Router, so
it's a documented exception to the rest of the package's polars/numpy-only
contract; gracefully degrades to a placeholder for non-MoE checkpoints.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth
diagnostics) with a lean giant/analysis/ package that compares one
autoregressive `giant rollout` for a checkpoint against a held-out
miniCaloSim reference file, and generates publication-styled plots in
parallel on HTCondor.
Rollout output and a raw reference file share a world-frame physical
column subset under identical names, so the old ALR/local-frame decode
machinery is gone — everything is world-frame mm/MeV.
- sources.py: canonical LazyFrames, synthetic-termination-row filtering,
the secondary view (rollout generation>0 tracks vs reference sec_*_list).
- reduce.py: streaming primitives — a single hist1d group_by pass, per-event
scalars, edep-weighted depth/transverse profiles, species share, leakage.
- context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material
group sets once into shared.json, so each compute job is one pass, no range
scan (histogram efficiency).
- catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg,
material}, per-event totals, shower profiles, species/leakage, secondaries.
- render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata.
- condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit):
one job per plot, compute/render split (workers polars-only, no LaTeX).
Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover
the reduce primitives, catalog id uniqueness + compute, condor submit, and a
guarded render smoke test. Delete the two predict-diagnostics notebooks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lars