5 Commits

Author	SHA1	Message	Date
lars	ffb7c0cc2a	Add rollout-quality distance, confusion, containment and router plots (gitea #76) ... Picks 4 of the 7 catalog additions the issue proposed (the smaller-lift ones; 2D joint plots, PIT calibration, and the throughput/accuracy scatter are left for follow-up issues): - marginal_distance_summary: a var x grouping-axis KS-statistic heatmap, reusing the existing marginal hist1d compute and just adding a finalize — a single at-a-glance regression scorecard instead of N overlay plots. - n_sec_confusion: predicted (rollout) vs true (reference) secondary count per event, paired by event_id since a rollout is seeded from the same events as its reference file. Needed a new zero-filling primitive (reduce.sec_count_by_event) since a plain group_by over secondary rows silently drops zero-secondary events. - shower_containment_depth_{90,95}: per-event depth containing 90%/95% of deposited energy, derived from the same per-event depth-bin matrix the longitudinal profile already computes. - router_specialization: max gate weight vs energy per side, summarizing router_gating's full stacked area into the one trend line the roadmap's MoE writeup describes (the ~60-65% ceiling), to make a future lambda_balance>0 retrain's effect on specialization checkable at a glance. Both new heatmap-shaped plots (distance summary, confusion matrix) share one new "heatmap" Reduced kind/renderer rather than two near-identical ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-24 11:12:16 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	86fc46b5a8	analyze: chunk per-plot aggregation across HTCondor jobs ... Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.	2026-07-27 09:24:19 +02:00
lars	60c2ca1985	analyze: add MoE router gating/share diagnostic plots ... New "model" family in the gallery: router_gating (mean soft gate weight vs. pre-step energy, showing the router's soft decision boundaries) and router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share by species / true physics process). Needs a live checkpoint's Router, so it's a documented exception to the rest of the package's polars/numpy-only contract; gracefully degrades to a placeholder for non-MoE checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:32:25 +02:00
lars	f4c2545e8b	Rewrite analysis as streaming rollout-vs-reference plotting pipeline ... Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth diagnostics) with a lean giant/analysis/ package that compares one autoregressive `giant rollout` for a checkpoint against a held-out miniCaloSim reference file, and generates publication-styled plots in parallel on HTCondor. Rollout output and a raw reference file share a world-frame physical column subset under identical names, so the old ALR/local-frame decode machinery is gone — everything is world-frame mm/MeV. - sources.py: canonical LazyFrames, synthetic-termination-row filtering, the secondary view (rollout generation>0 tracks vs reference sec_*_list). - reduce.py: streaming primitives — a single hist1d group_by pass, per-event scalars, edep-weighted depth/transverse profiles, species share, leakage. - context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material group sets once into shared.json, so each compute job is one pass, no range scan (histogram efficiency). - catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg, material}, per-event totals, shower profiles, species/leakage, secondaries. - render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata. - condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit): one job per plot, compute/render split (workers polars-only, no LaTeX). Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover the reduce primitives, catalog id uniqueness + compute, condor submit, and a guarded render smoke test. Delete the two predict-diagnostics notebooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:38:09 +02:00