Allow giant analyze to compare multiple rollouts against a single shared reference #77

New Issue

2026-08-24T10:28:41+02:00

lars commented

2026-08-24 10:28:41 +02:00

Today giant analyze is hardwired end-to-end for exactly one rollout vs one reference. Goal:
accept multiple rollouts, render them as distinct colored series in the same plots, with
exactly one reference line/histogram — and assert all supplied rollouts point at the same
reference (seed) file, since the whole premise is "compare candidates against one ground
truth," not N independent comparisons.

Concrete places this is currently singular:

giant/analysis/sources.py — Side enum (line ~84) is binary rollout/reference.
giant/analysis/catalog.py — Bundle (line ~65) holds one fixed
r_all/t_all/r_phys/t_phys pair; the literal keys _ROLL = "rollout" / _REF = "reference"
(line ~147) are baked into every compute_partial/finalize across all plot families
(marginals, per-event totals, shower profiles, species/leakage, secondaries, router
diagnostics).
giant/analysis/reduced.py — Reduced.payload is keyed by the string literals
"rollout"/"reference" (or "{key}_mean"/"_std" for profiles), not a list/dict of series.
giant/analysis/render.py — _SERIES_LABELS (line ~25) is a fixed 2-entry dict; every
renderer (_render_overlay, _render_profile, _render_bar, _render_router_gating,
_render_router_share) hardcodes iteration over ("reference","rollout") or the reverse.
No explicit color map exists — colors come from matplotlib's default cycle over exactly 2
items. _render_bar's bar-offset math (width/2) assumes exactly 2 bars.
giant/analysis/condor.py — load_rollout_yaml() (line ~100) requires exactly one YAML
(output + dataset keys); RunMeta.rollout: str (line ~144) is singular; derive_run_dir,
prep(), compute_reduced/compute_one all assume one rollout. The condor job grid is
chunked only by (plot_id, chunk_index) — no rollout dimension.
giant/cli.py — analyze prep/analyze submit (analyze_app, ~line 1550) take a single
positional rollout_yaml: Path; analyze render takes a single run_dir.

Needed changes:

CLI: accept multiple rollout YAMLs (e.g. giant analyze submit rollout_a.yaml rollout_b.yaml ...). Validate up front that every YAML's dataset (reference file) matches
— fail fast with a clear error otherwise. Single-rollout usage (today's YAML schema, N=1)
must keep working unchanged.
sources.py: generalize the binary Side into "one reference side" + "a named list of
rollout sides" (name = YAML stem, or an explicit label field in the YAML).
catalog.py: generalize Bundle to a reference pair plus a dict[name, frames] for
rollouts; rewrite every compute_partial/finalize to loop over rollout names instead of
the hardcoded "rollout" key.
reduced.py: change Reduced.payload schema so series are represented as a
name-keyed structure (e.g. payload["series"] = {name: ...}) with the reference kept as one
distinguished entry, not a second hardcoded literal.
render.py: replace _SERIES_LABELS and the hardcoded 2-tuples with a loop over N rollout
series + 1 reference; add an explicit color palette (reference gets one fixed, consistent
style — e.g. black/dashed — across every plot; each rollout gets a distinct color, stable
across plots for the same run). Fix _render_bar's two-bar offset math to handle N+1 bars.
condor.py: RunMeta.rollout: str → a list of named rollouts sharing one reference;
derive_run_dir needs to combine multiple rollout identifiers into one run-dir name;
compute_reduced/compute_one need a way to iterate rollouts per (plot, chunk) job —
likely simplest to keep chunking by event_id and loop rollouts inside one compute-one
job (reading N rollout files instead of 1), rather than exploding the condor queue by a
rollout dimension.
Note this changes run-directory identity: today a run dir is keyed by one rollout's
prediction_id/stem; with N rollouts it needs to represent a set.

Today `giant analyze` is hardwired end-to-end for exactly one rollout vs one reference. Goal: accept multiple rollouts, render them as distinct colored series in the same plots, with exactly one reference line/histogram — and assert all supplied rollouts point at the same reference (seed) file, since the whole premise is "compare candidates against one ground truth," not N independent comparisons. Concrete places this is currently singular: - `giant/analysis/sources.py` — `Side` enum (line ~84) is binary `rollout`/`reference`. - `giant/analysis/catalog.py` — `Bundle` (line ~65) holds one fixed `r_all/t_all/r_phys/t_phys` pair; the literal keys `_ROLL = "rollout"` / `_REF = "reference"` (line ~147) are baked into every `compute_partial`/`finalize` across all plot families (marginals, per-event totals, shower profiles, species/leakage, secondaries, router diagnostics). - `giant/analysis/reduced.py` — `Reduced.payload` is keyed by the string literals `"rollout"`/`"reference"` (or `"{key}_mean"/"_std"` for profiles), not a list/dict of series. - `giant/analysis/render.py` — `_SERIES_LABELS` (line ~25) is a fixed 2-entry dict; every renderer (`_render_overlay`, `_render_profile`, `_render_bar`, `_render_router_gating`, `_render_router_share`) hardcodes iteration over `("reference","rollout")` or the reverse. No explicit color map exists — colors come from matplotlib's default cycle over exactly 2 items. `_render_bar`'s bar-offset math (`width/2`) assumes exactly 2 bars. - `giant/analysis/condor.py` — `load_rollout_yaml()` (line ~100) requires exactly one YAML (`output` + `dataset` keys); `RunMeta.rollout: str` (line ~144) is singular; `derive_run_dir`, `prep()`, `compute_reduced`/`compute_one` all assume one rollout. The condor job grid is chunked only by `(plot_id, chunk_index)` — no rollout dimension. - `giant/cli.py` — `analyze prep`/`analyze submit` (`analyze_app`, ~line 1550) take a single positional `rollout_yaml: Path`; `analyze render` takes a single `run_dir`. Needed changes: 1. CLI: accept multiple rollout YAMLs (e.g. `giant analyze submit rollout_a.yaml rollout_b.yaml ...`). Validate up front that every YAML's `dataset` (reference file) matches — fail fast with a clear error otherwise. Single-rollout usage (today's YAML schema, N=1) must keep working unchanged. 2. `sources.py`: generalize the binary `Side` into "one reference side" + "a named list of rollout sides" (name = YAML stem, or an explicit label field in the YAML). 3. `catalog.py`: generalize `Bundle` to a reference pair plus a `dict[name, frames]` for rollouts; rewrite every `compute_partial`/`finalize` to loop over rollout names instead of the hardcoded `"rollout"` key. 4. `reduced.py`: change `Reduced.payload` schema so series are represented as a name-keyed structure (e.g. `payload["series"] = {name: ...}`) with the reference kept as one distinguished entry, not a second hardcoded literal. 5. `render.py`: replace `_SERIES_LABELS` and the hardcoded 2-tuples with a loop over N rollout series + 1 reference; add an explicit color palette (reference gets one fixed, consistent style — e.g. black/dashed — across every plot; each rollout gets a distinct color, stable across plots for the same run). Fix `_render_bar`'s two-bar offset math to handle N+1 bars. 6. `condor.py`: `RunMeta.rollout: str` → a list of named rollouts sharing one `reference`; `derive_run_dir` needs to combine multiple rollout identifiers into one run-dir name; `compute_reduced`/`compute_one` need a way to iterate rollouts per (plot, chunk) job — likely simplest to keep chunking by `event_id` and loop rollouts *inside* one compute-one job (reading N rollout files instead of 1), rather than exploding the condor queue by a rollout dimension. 7. Note this changes run-directory identity: today a run dir is keyed by one rollout's `prediction_id`/stem; with N rollouts it needs to represent a set.

lars added the modularity feature labels 2026-08-24 10:28:41 +02:00

lars referenced this issue from a commit

2026-08-24 13:23:59 +02:00

Add multi-rollout support to giant analyze (gitea #77)

lars referenced this issue

2026-08-24 13:25:30 +02:00

Add multi-rollout support to giant analyze (gitea #77) #80

lars referenced this issue

2026-08-24 13:25:57 +02:00

shower_containment_depth_90/95 title crashes LaTeX rendering (unescaped %) #81

lars referenced this issue from a commit

2026-08-24 14:33:08 +02:00