4 Commits

Author	SHA1	Message	Date
lars	ffb7c0cc2a	Add rollout-quality distance, confusion, containment and router plots (gitea #76) ... Picks 4 of the 7 catalog additions the issue proposed (the smaller-lift ones; 2D joint plots, PIT calibration, and the throughput/accuracy scatter are left for follow-up issues): - marginal_distance_summary: a var x grouping-axis KS-statistic heatmap, reusing the existing marginal hist1d compute and just adding a finalize — a single at-a-glance regression scorecard instead of N overlay plots. - n_sec_confusion: predicted (rollout) vs true (reference) secondary count per event, paired by event_id since a rollout is seeded from the same events as its reference file. Needed a new zero-filling primitive (reduce.sec_count_by_event) since a plain group_by over secondary rows silently drops zero-secondary events. - shower_containment_depth_{90,95}: per-event depth containing 90%/95% of deposited energy, derived from the same per-event depth-bin matrix the longitudinal profile already computes. - router_specialization: max gate weight vs energy per side, summarizing router_gating's full stacked area into the one trend line the roadmap's MoE writeup describes (the ~60-65% ceiling), to make a future lambda_balance>0 retrain's effect on specialization checkable at a glance. Both new heatmap-shaped plots (distance summary, confusion matrix) share one new "heatmap" Reduced kind/renderer rather than two near-identical ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-24 11:12:16 +02:00
lars	c1e6ffd8c6	Offset event_id across multi-shard reference reads in giant analyze (gitea #22) ... giant/analysis/sources.py's open_side scanned a reference directory of parquet shards with a bare glob and never offset event_id across them. Each shard is a separate Geant4 job whose own event_id numbering restarts from 0, so events from different shards collided on the same event_id, corrupting every downstream per-event grouping and the event_id % n_chunks condor chunking — the same root cause already fixed on the training/rollout side via giant/data/loader.py's per-file event_id_offset. open_side's reference branch now uses find_parquet_files (the same deterministically ordered file lister giant rollout's _seed_from_data uses) and offsets each shard's event_id via a join on polars' include_file_paths, so both sides of a comparison agree on what an event_id means. Two incidental behaviour changes come along for free: .manifest references now work (they crashed before), and the directory glob narrows from recursive **/*.parquet to top-level *.parquet, matching the file list rollout itself used to assign offsets — a deliberate choice, since a differing file list would make the two sides' offsets disagree again in a subtler way. No overflow guard on the per-shard offset stride (unlike loader's _offset_event_id): checking it here would cost an eager event_id-column read per shard in every condor compute job, and giant rollout already runs that check over the same file list when producing the seed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-17 15:53:02 +02:00
lars	a746efb6e1	Clamp analysis histogram bins before the i32 cast, not after (gitea #61) ... _bin_expr in giant/analysis/reduce.py clipped the bin index to [0, nbins-1] only after casting it to Int32, so the clip never got a chance to run: a rollout step_length of 1.0725e10 mm against fixed edges [2.9e-5, 94.04] with 50 bins produces a raw index of ~5.7e9, which overflows i32 and fails the strict cast, killing the whole compute-one job. Same failure mode for +/-inf. Clamp in f64 first, then cast to Int32. NaN has no edge to clamp to, so it maps to null and is dropped in the two callers (hist1d, profile_partial) — matching what np.histogram does with NaN, and what profile_partial needs anyway since a null bin index would break its np.add.at. This reimplements commit 313373c, which fixed the same bug but landed on a branch (fix/rollout-negative-secondary-mass) that forked off a stale master and was never merged; reduce.py has since diverged enough that the original diff no longer applies cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 09:40:09 +02:00
lars	f4c2545e8b	Rewrite analysis as streaming rollout-vs-reference plotting pipeline ... Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth diagnostics) with a lean giant/analysis/ package that compares one autoregressive `giant rollout` for a checkpoint against a held-out miniCaloSim reference file, and generates publication-styled plots in parallel on HTCondor. Rollout output and a raw reference file share a world-frame physical column subset under identical names, so the old ALR/local-frame decode machinery is gone — everything is world-frame mm/MeV. - sources.py: canonical LazyFrames, synthetic-termination-row filtering, the secondary view (rollout generation>0 tracks vs reference sec_*_list). - reduce.py: streaming primitives — a single hist1d group_by pass, per-event scalars, edep-weighted depth/transverse profiles, species share, leakage. - context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material group sets once into shared.json, so each compute job is one pass, no range scan (histogram efficiency). - catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg, material}, per-event totals, shower profiles, species/leakage, secondaries. - render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata. - condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit): one job per plot, compute/render split (workers polars-only, no LaTeX). Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover the reduce primitives, catalog id uniqueness + compute, condor submit, and a guarded render smoke test. Delete the two predict-diagnostics notebooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:38:09 +02:00