Refactor the analysis plot creation with focus on rollout #16

Merged

lars merged 30 commits from analysis-rollout-plots into master 2026-07-27 14:58:51 +02:00

Conversation 0 Commits 30 Files Changed 36

+4582 -5007

lars commented 2026-07-24 13:51:33 +02:00

Owner

Copy Link

Copy Source

No description provided.

lars added 8 commits 2026-07-24 13:51:34 +02:00

Rewrite analysis as streaming rollout-vs-reference plotting pipeline ... f4c2545e8b

Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth
diagnostics) with a lean giant/analysis/ package that compares one
autoregressive `giant rollout` for a checkpoint against a held-out
miniCaloSim reference file, and generates publication-styled plots in
parallel on HTCondor.

Rollout output and a raw reference file share a world-frame physical
column subset under identical names, so the old ALR/local-frame decode
machinery is gone — everything is world-frame mm/MeV.

- sources.py: canonical LazyFrames, synthetic-termination-row filtering,
  the secondary view (rollout generation>0 tracks vs reference sec_*_list).
- reduce.py: streaming primitives — a single hist1d group_by pass, per-event
  scalars, edep-weighted depth/transverse profiles, species share, leakage.
- context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material
  group sets once into shared.json, so each compute job is one pass, no range
  scan (histogram efficiency).
- catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg,
  material}, per-event totals, shower profiles, species/leakage, secondaries.
- render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata.
- condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit):
  one job per plot, compute/render split (workers polars-only, no LaTeX).

Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover
the reduce primitives, catalog id uniqueness + compute, condor submit, and a
guarded render smoke test. Delete the two predict-diagnostics notebooks.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

analyze: drive prep/submit from the rollout YAML sidecar ... 8ff70e3c87

`giant analyze prep` / `submit` now take the `giant rollout` YAML sidecar as
their only positional input instead of explicit --rollout/--reference/--out-dir.
The YAML's `output`/`dataset` keys name the rollout parquet and its seed file
(the reference truth), and the rest of the sidecar (checkpoint, geometry oracle,
cutoffs) flows into every plot's gallery metadata.

prep derives its own run directory next to the rollout parquet
(<...>/analysis_<id>/) holding shared.json, run_meta.json, reduced/, plots/.
compute-one and render now take just --run-dir / a run-dir argument and read the
resolved paths + metadata from run_meta.json, so the condor wrapper no longer
threads file paths. open_side scans a directory of reference shards via glob.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

analyze: show model/training params on rendered figures ... 1e7d7d7efd

`giant rollout` now records the checkpoint's architecture (mode,
hidden_dim, n_blocks, emb_dim, dropout, conditioning) plus training_epoch
and best_val_loss in its YAML sidecar, using data already loaded from the
checkpoint. condor.py carries those through run_meta, and render.py passes
them to plotstyle's new_figure(params=...) so every plot's subtitle shows
what produced it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

deps: install plotstyle from git.larsbogner.de package index ... 6e1d11b8d8

Swap the local/editable ETPlot `gallery[plotting]` dependency for
`plotstyle>=1.0.0`, now published to a package registry, so the repo
doesn't need a local ETPlot checkout to resolve.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

analyze: drop stale ty:ignore on plotstyle import ... 77b2e9e5f4

plotstyle is now a real installed package (from the git.larsbogner.de
index) with its own type info, so the unresolved-import suppression
is no longer needed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

test: replace prep(**_CTX) splat with a typed _prep helper ... 1115301eb9

ty correctly flagged this as unsound: _CTX's inferred dict[str, int]
type doesn't rule out a "run_dir" key, which would silently bind to
prep's own run_dir: str | Path | None parameter instead of falling
through to **ctx_kwargs. Passing the context kwargs by name in a
small test helper removes the ambiguity.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

analyze: add MoE router gating/share diagnostic plots ... 60c2ca1985

New "model" family in the gallery: router_gating (mean soft gate weight
vs. pre-step energy, showing the router's soft decision boundaries) and
router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share
by species / true physics process). Needs a live checkpoint's Router, so
it's a documented exception to the rest of the package's polars/numpy-only
contract; gracefully degrades to a placeholder for non-MoE checkpoints.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

chore: remove stray CUDA sanity script and stale Phase 2 planning doc ... b6893b0118

test-cuda.py was a one-off local CUDA check, not part of tests/ or scripts/.
docs/phase2_plan.md is superseded by the "Phase 2 (implemented)" section of
CLAUDE.md's roadmap.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-24 14:20:27 +02:00

docs: document compute environment, WGAN/MoE status, and condor-gpu-train-rollout ... 80d544aa73

Adds a Compute environment section (laptop/desktop vs. shared portal
machines vs. condor workers) and corrects the roadmap: the WGAN-GP and
MoE routing-trunk tracks are actually implemented (untested and
under-testing respectively), not "not yet built" as previously stated.
Also notes the in-progress condor-gpu-train-rollout branch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-24 14:32:10 +02:00

analyze: normalize pdg dtype in open_side to fix rollout/reference concat ... 70d0f04326

Rollout output and the reference file's ROOT-derived parquet disagree on
pdg's integer width (Int32 vs Int64), which only surfaced downstream as a
pl.concat SchemaError in build_context's pdg-count merge. Cast to a
canonical Int64 at the single scan entry point instead.

lars added 1 commit 2026-07-27 09:24:54 +02:00

analyze: chunk per-plot aggregation across HTCondor jobs ... 86fc46b5a8

Add a second parallelism axis to giant analyze: each plot's data can now
be split into a configurable number of event_id-disjoint chunks, each
computed as its own HTCondor job, bounding per-job walltime and scan cost
on large rollout/reference files instead of one job re-scanning the
whole file per plot.

Every PlotSpec now splits into compute_partial (runs per (plot, chunk)
job against a chunk-filtered Bundle) and finalize (merges chunks -
elementwise sum for fixed-edge histograms/species shares, concatenate
-then-recompute for specs that derive edges or mean/std from the full
per-event/per-secondary array). Router diagnostics stay chunkable=False
and always run as a single job. giant analyze render now joins every
plot's chunk partials (merge_all) before rendering, transparently.

New: --chunks on `analyze prep`/`analyze submit`, --chunk on
`analyze compute-one`, and a new `analyze merge-one` command.

lars added 1 commit 2026-07-27 09:30:59 +02:00

analyze: expose bin/pdg options on analyze submit ... 85d3914a4d

`submit` calls `prep` internally but only forwarded --chunks, so a
condor run could never use non-default energy-bins/bins/top-pdg.

lars added 1 commit 2026-07-27 09:47:48 +02:00

analyze: estimate per-job HTCondor walltime from chunk row count ... e380400fe9

Each condor job's +RequestWalltime used to be one flat 3600s default
for every (plot, chunk), regardless of how much data it actually
streams over. `prep` now records each chunk's rollout+reference row
count, and `giant/analysis/runtime_estimate.py` turns that into a
per-job estimate: a per-spec (intercept, seconds/row) cost model fit
by `scripts/profile_analysis_costs.py` against synthetic mock data on
this machine, plus a fixed overhead placeholder (docker/uv/shared-fs
startup — unmeasurable here, no /ceph access) and a single
RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and
the submit description references it via $(walltime) instead of a
constant.

lars added 1 commit 2026-07-27 10:03:08 +02:00

analyze: run condor compute jobs via .venv/bin/giant, not uv run ... fa59443339

uv isn't installed on the HTCondor worker docker image, so `uv run`
fails there. giant is already an installed console script in the
repo's uv-synced .venv, so exec it directly instead. write_submit now
fails fast with a clear message if .venv/bin/giant is missing.

lars added 1 commit 2026-07-27 10:14:17 +02:00

analyze: default condor docker image to alma9-gridjob ... dbd5c7e083

mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current
numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9
not found). Switch the default to cverstege/alma9-gridjob, a modern
EL9-based image.

lars added 1 commit 2026-07-27 10:31:49 +02:00

analyze: raise default condor job memory request to 8192 MB ... c7194701f6

4096 MB was too tight: a resubmitted run held ~31 jobs spread evenly
across nearly every plot family and chunk index with "Docker job has
gone over memory limit of 4224 Mb", not one specific spec, so the
generic per-chunk data footprint needed more headroom.

lars added 2 commits 2026-07-27 11:06:53 +02:00

analyze: recalibrate condor walltime model from real cluster timings ... 4d6101dcd7

The prior _COST_MODEL/_FIXED_OVERHEAD_S were fit only against local
synthetic benchmarks (up to 2M rows/side), which can't see docker
pull or real /ceph read latency and wildly overestimated real jobs
(~1200-1800s predicted vs 50-320s median actual, from condor_history
on production run 563f5ee3, --chunks 4, ~254M total rows).

Refit each spec's per-row rate through the origin against its median
real wall-clock time (not max, to avoid baking a few /ceph-contention
spikes into a rate that would then wrongly scale with dataset size),
and raised RUNTIME_SAFETY_MARGIN to compensate for that same
contention risk instead.

transforms: pad legacy cond normalizers for pre-physical-conditioning checkpoints ... eb751d968d

Checkpoints trained before commit 68fb99b (physical-property
conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond
normalizer, fit before build_cond_features grew the extra physical
columns. Any inference against such a checkpoint under current code
(predict/rollout/router_gating) crashed broadcasting a 15-wide
cond_cont against an 8-wide mean/std.

In "embedding" mode those physical columns are never read by
ConditionEncoder, so padding the missing entries with mean=0/std=1 is
a safe no-op. "physical" mode reads them directly, so a mismatch there
still raises instead of silently normalizing garbage.

lars added 1 commit 2026-07-27 11:19:08 +02:00

style: ruff format a88b21ef70

lars added 1 commit 2026-07-27 11:29:37 +02:00

analyze: default run directory to <repo>/analysis_runs, gitignored ... 61410ddee3

giant analyze prep/submit previously defaulted the run directory to
next to the rollout parquet on /ceph. Default it instead to
<cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo
checkout (/work) — gitignored, --run-dir still overrides it.
derive_run_dir/prep gained a default_base param; library callers that
don't pass one keep the old parquet-relative fallback.

lars added 1 commit 2026-07-27 12:06:48 +02:00

docs: record first MoE router rollout benchmark result in the roadmap ... 641bbb0a68

The 10-expert EnergyRouter checkpoint (lambda_balance=0.0) diverges badly
from Geant4 on rollout (step granularity, secondary species, shower
shape), and the router_gating diagnostic shows the experts heavily
overlap rather than partitioning the pre-step energy domain. Recorded as
"needs retraining with a different router config" rather than an
abandoned direction. Full analysis in the knowledge base at
experiments/giant-router-energy-rollout-validation.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-27 13:11:18 +02:00

router: seed EnergyRouter centers from data quantiles instead of a fixed linspace ... ee29b9a303

The 2026-07-22 rollout benchmark's router_gating diagnostic showed the
10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a
roughly uniform z-normalized energy distribution, leaving experts heavily
overlapping instead of partitioning the range. Add an optional
centers_init kwarg (backward compatible, defaults to the old linspace) and
have giant train estimate it from a reservoir sample of the real energy
column, collected during the existing normalizer-fitting pass.

lars added 2 commits 2026-07-27 13:35:08 +02:00

docs: note the EnergyRouter centers_init fix in the roadmap ... 4feb9007f3

Records both identified contributors to the 2026-07-22 router divergence
(missing lambda_balance and the uniform-linspace center init) and that
the latter is now fixed, with next steps covering both for the retrain.

analyze: thread full model/training/rollout/dataset params to plots ... b8a4dc7d63

giant rollout now writes the checkpoint's complete model_config (incl.
the router sub-dict), the sibling config.toml's [train]/[meta] sections,
and every rollout CLI knob (weights, batch_size, escape_threshold,
n_events, device, seed) into the YAML sidecar instead of a hand-picked
subset. All of it flows through run_meta.json into each plot's own
metadata.yaml for later comparison, while the figure subtitle itself
shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router,
epoch, best_val_loss, steps/noise_dim) via new_figure's params option.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-27 14:01:05 +02:00

ci: share one uv sync across jobs, gate tests on lint+type-check, sync tag/version on release tags ... 08c76a9614

Uploads the synced .venv as an artifact from a single setup job instead of
re-running uv sync (and re-downloading torch) in every job. Drops the
build/publish job in favor of a lighter job that, on a pushed tag, checks the
tag against the uv project version and — if they differ — bumps the version,
commits it to master, and recreates the tag on the new commit.

lars added 1 commit 2026-07-27 14:18:59 +02:00

ci: replace unsupported artifact sharing with a bind-mounted uv cache ... 0afa75ee30

The self-hosted act_runner doesn't support upload/download-artifact, so drop
that plumbing and instead run jobs in an explicit container with a persistent
host directory mounted at /uv-cache (UV_CACHE_DIR), backed by valid_volumes on
the runner. uv sync still runs per job but hits a warm local cache instead of
re-downloading/building packages every time.

lars added 1 commit 2026-07-27 14:22:56 +02:00

ci: stop setup-uv from overriding UV_CACHE_DIR ... 1787d26d1c

setup-uv's default enable-cache: auto sets its own UV_CACHE_DIR (a tool-cache
tmp path) as a later step, clobbering the workflow-level UV_CACHE_DIR that
points at the bind-mounted persistent cache. Disable setup-uv's own cache
handling so our mount stays in effect.

lars added 1 commit 2026-07-27 14:31:28 +02:00

ci: re-pin UV_CACHE_DIR after setup-uv, which exports its own value regardless of enable-cache ... 19be455346

Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path
(/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow
env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step
right after setup-uv so it wins for the rest of the job.

lars added 1 commit 2026-07-27 14:40:05 +02:00

ci: set UV_LINK_MODE=copy to silence the cross-filesystem hardlink warning ... 0f7febfd02

The bind-mounted uv cache and the job workspace are on different
filesystems, so uv already falls back to copying installed files; this just
tells it to do so directly instead of logging a hardlink-failed warning
every job.

lars marked the pull request as ready for review 2026-07-27 14:49:20 +02:00

lars added 1 commit 2026-07-27 14:55:03 +02:00

Merge remote-tracking branch 'origin/master' into analysis-rollout-plots ... 5594866eac

# Conflicts:
#	analysis/rollout_validation.ipynb

lars merged commit 6208810089 into master 2026-07-27 14:58:51 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

experiment

investigation

performance

phase2

validation

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

lars (Lars Bogner)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#16