Add new rollout-quality plots to the analysis catalog #76

New Issue

2026-08-24T10:28:37+02:00

lars commented

2026-08-24 10:28:37 +02:00

giant/analysis/catalog.py's current PlotSpec registry is all 1D: marginals (× overall/
energy/pdg/material), per-event totals, shower profiles, species/leakage, secondaries, and
router diagnostics. These are good for eyeballing agreement but don't give a single
quantitative "did this checkpoint get better" score, don't check calibration, and don't check
correlations between variables — a rollout could pass every 1D marginal while still being wrong
jointly.

Proposal list (not all needed at once — pick and triage into sub-issues as work starts):

Quantitative distance summary panel. For each existing marginal (across
MARGINAL_VARS × group axis), compute a scalar distance (Wasserstein-1 or KS statistic)
between rollout and reference histograms and render as a var × group heatmap/table instead
of N separate overlay plots. Gives an at-a-glance regression scorecard, and could gate
whether a new checkpoint is "better than baseline." Reuses the hist1d binning already
computed for marginals — mostly a new finalize().
2D correlation / joint plots. E.g. step_length vs edep, or scattering angle vs pre_E,
as side-by-side 2D histograms (or a ratio/difference heatmap) for rollout vs reference. The
pipeline is 1D-only today, so correlated errors (e.g. energy conservation holding per-axis
but the joint shape being wrong) are currently invisible. Needs a new reduce.py streaming
primitive (2D binned group_by).
Rank/PIT calibration histogram. Draw M stochastic rollout samples per held-out
conditioning vector, and compute the rank of the single reference truth within the ensemble
(probability integral transform). A flat rank histogram = well-calibrated; U-shaped/humped
reveals under/overdispersion. Standard generative-model diagnostic, and a real probe of
model quality beyond marginal agreement — but needs giant rollout to support multiple
stochastic draws per condition, which it doesn't today. Larger lift; flag as such.
n_sec confusion matrix. 2D heatmap of predicted vs true n_sec (both already
available via the secondaries family) instead of two separate 1D marginals — shows
directly whether errors are systematic (e.g. always undercounting) vs random noise.
Shower containment-depth distribution. Per-event derived quantity — e.g. depth
containing 90%/95% of deposited energy — rollout vs reference. More discriminating for
shower-shape realism than per-step marginals, similar shape to the existing event family.
Eval throughput vs. accuracy tradeoff. Scatter of the distance score from (1) against
measured shower/sec throughput, one point per mode (flow/ddpm/wgan/router) or checkpoint.
Ties directly to the CLAUDE.md roadmap's "faster-eval architectures... neither has a
validated result yet" — gives one plot answering "is the speed win worth the accuracy cost."
Router specialization score over energy. Extend the existing router_gating plot with
a scalar summary curve (max gate weight, or normalized entropy) vs energy, making the
~60-65% gate-weight ceiling described in the roadmap's MoE writeup visible as a single trend
line rather than requiring visual inspection of the full gating heatmap — useful for
confirming whether a retrain with lambda_balance > 0 actually sharpened specialization.

`giant/analysis/catalog.py`'s current `PlotSpec` registry is all 1D: marginals (× overall/ energy/pdg/material), per-event totals, shower profiles, species/leakage, secondaries, and router diagnostics. These are good for eyeballing agreement but don't give a single quantitative "did this checkpoint get better" score, don't check calibration, and don't check correlations between variables — a rollout could pass every 1D marginal while still being wrong jointly. Proposal list (not all needed at once — pick and triage into sub-issues as work starts): 1. **Quantitative distance summary panel.** For each existing marginal (across `MARGINAL_VARS` × group axis), compute a scalar distance (Wasserstein-1 or KS statistic) between rollout and reference histograms and render as a var × group heatmap/table instead of N separate overlay plots. Gives an at-a-glance regression scorecard, and could gate whether a new checkpoint is "better than baseline." Reuses the `hist1d` binning already computed for marginals — mostly a new `finalize()`. 2. **2D correlation / joint plots.** E.g. step_length vs edep, or scattering angle vs pre_E, as side-by-side 2D histograms (or a ratio/difference heatmap) for rollout vs reference. The pipeline is 1D-only today, so correlated errors (e.g. energy conservation holding per-axis but the joint shape being wrong) are currently invisible. Needs a new `reduce.py` streaming primitive (2D binned `group_by`). 3. **Rank/PIT calibration histogram.** Draw M stochastic rollout samples per held-out conditioning vector, and compute the rank of the single reference truth within the ensemble (probability integral transform). A flat rank histogram = well-calibrated; U-shaped/humped reveals under/overdispersion. Standard generative-model diagnostic, and a real probe of model quality beyond marginal agreement — but needs `giant rollout` to support multiple stochastic draws per condition, which it doesn't today. Larger lift; flag as such. 4. **n_sec confusion matrix.** 2D heatmap of predicted vs true `n_sec` (both already available via the `secondaries` family) instead of two separate 1D marginals — shows directly whether errors are systematic (e.g. always undercounting) vs random noise. 5. **Shower containment-depth distribution.** Per-event derived quantity — e.g. depth containing 90%/95% of deposited energy — rollout vs reference. More discriminating for shower-shape realism than per-step marginals, similar shape to the existing `event` family. 6. **Eval throughput vs. accuracy tradeoff.** Scatter of the distance score from (1) against measured shower/sec throughput, one point per mode (flow/ddpm/wgan/router) or checkpoint. Ties directly to the CLAUDE.md roadmap's "faster-eval architectures... neither has a validated result yet" — gives one plot answering "is the speed win worth the accuracy cost." 7. **Router specialization score over energy.** Extend the existing `router_gating` plot with a scalar summary curve (max gate weight, or normalized entropy) vs energy, making the ~60-65% gate-weight ceiling described in the roadmap's MoE writeup visible as a single trend line rather than requiring visual inspection of the full gating heatmap — useful for confirming whether a retrain with `lambda_balance > 0` actually sharpened specialization.

lars added the validation feature labels 2026-08-24 10:28:37 +02:00

lars referenced this issue from a commit

2026-08-24 11:12:24 +02:00

Add rollout-quality distance, confusion, containment and router plots (gitea #76)

lars referenced this issue

2026-08-24 11:14:05 +02:00

Add rollout-quality distance, confusion, containment and router plots (gitea #76) #79

lars referenced this issue from a commit

2026-08-24 11:22:14 +02:00

Merge pull request 'Add rollout-quality distance, confusion, containment and router plots (gitea #76)' (#79) from fix/issue-76 into master

lars closed this issue

2026-08-24 11:44:03 +02:00