6 Commits

Author	SHA1	Message	Date
lars	ebd3e0dc71	Add multi-rollout support to giant analyze (gitea #77) ... giant analyze compares N rollout YAMLs against one shared reference file (all must name the same dataset, checked up front) instead of exactly one rollout vs one reference, rendering each rollout as its own colored series against a single reference line/panel. Series names come from a repeated --label flag, else the YAML stem, else "rollout" for a single YAML — a single-rollout run keeps rendering identically to before this change. Bundle now holds a name-keyed dict of rollout sides instead of one fixed pair, every catalog compute_partial/finalize builds a Reduced.payload keyed the same way ("series": {name: ...}, "reference": ... as the one distinguished non-rollout entry), and every renderer draws N series (or N panels, for the two heatmap-shaped specs and the router/type-embedding diagnostics, which are inherently one-matrix/one-checkpoint per rollout) against the reference's fixed dashed-ink style.	2026-08-24 13:23:50 +02:00
lars	ffb7c0cc2a	Add rollout-quality distance, confusion, containment and router plots (gitea #76) ... Picks 4 of the 7 catalog additions the issue proposed (the smaller-lift ones; 2D joint plots, PIT calibration, and the throughput/accuracy scatter are left for follow-up issues): - marginal_distance_summary: a var x grouping-axis KS-statistic heatmap, reusing the existing marginal hist1d compute and just adding a finalize — a single at-a-glance regression scorecard instead of N overlay plots. - n_sec_confusion: predicted (rollout) vs true (reference) secondary count per event, paired by event_id since a rollout is seeded from the same events as its reference file. Needed a new zero-filling primitive (reduce.sec_count_by_event) since a plain group_by over secondary rows silently drops zero-secondary events. - shower_containment_depth_{90,95}: per-event depth containing 90%/95% of deposited energy, derived from the same per-event depth-bin matrix the longitudinal profile already computes. - router_specialization: max gate weight vs energy per side, summarizing router_gating's full stacked area into the one trend line the roadmap's MoE writeup describes (the ~60-65% ceiling), to make a future lambda_balance>0 retrain's effect on specialization checkable at a glance. Both new heatmap-shaped plots (distance summary, confusion matrix) share one new "heatmap" Reduced kind/renderer rather than two near-identical ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-24 11:12:16 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	86fc46b5a8	analyze: chunk per-plot aggregation across HTCondor jobs ... Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.	2026-07-27 09:24:19 +02:00
lars	60c2ca1985	analyze: add MoE router gating/share diagnostic plots ... New "model" family in the gallery: router_gating (mean soft gate weight vs. pre-step energy, showing the router's soft decision boundaries) and router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share by species / true physics process). Needs a live checkpoint's Router, so it's a documented exception to the rest of the package's polars/numpy-only contract; gracefully degrades to a placeholder for non-MoE checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:32:25 +02:00
lars	f4c2545e8b	Rewrite analysis as streaming rollout-vs-reference plotting pipeline ... Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth diagnostics) with a lean giant/analysis/ package that compares one autoregressive `giant rollout` for a checkpoint against a held-out miniCaloSim reference file, and generates publication-styled plots in parallel on HTCondor. Rollout output and a raw reference file share a world-frame physical column subset under identical names, so the old ALR/local-frame decode machinery is gone — everything is world-frame mm/MeV. - sources.py: canonical LazyFrames, synthetic-termination-row filtering, the secondary view (rollout generation>0 tracks vs reference sec_*_list). - reduce.py: streaming primitives — a single hist1d group_by pass, per-event scalars, edep-weighted depth/transverse profiles, species share, leakage. - context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material group sets once into shared.json, so each compute job is one pass, no range scan (histogram efficiency). - catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg, material}, per-event totals, shower profiles, species/leakage, secondaries. - render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata. - condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit): one job per plot, compute/render split (workers polars-only, no LaTeX). Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover the reduce primitives, catalog id uniqueness + compute, condor submit, and a guarded render smoke test. Delete the two predict-diagnostics notebooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:38:09 +02:00