Fits normalizers over multi-hundred-million-row datasets, so the setup
pass's per-row Python overhead compounds fast: encode_secondaries
recomputed an O(K) prefix sum from scratch on every one of its 15
stick-breaking iterations, np.isin re-sorted the full train-event-id
array on every chunk, and pdg/material/process index lookups ran a
Python dict lookup per row. The normalizer-fit pass also computed
encode_secondaries's stick-logit and direction-rotation blocks in full
even though it only ever reads the mass/charge columns.
Replace the prefix-sum recompute with a single np.cumsum, add a
sorted_membership helper (searchsorted-based) in place of np.isin at
both the setup-pass and per-epoch call sites, vectorize the index
lookups via _vectorized_map_lookup, and add an opt-in phys_only path
so the setup pass skips the stick-breaking/rotation work it discards
anyway. All four changes are output-identical performance refactors,
backed by new unit tests plus the existing suite.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The per-dim print loop lacked the empty-array guard already used for
the KL computation right above it and the sec-slot loop further down,
so an all-zero-secondaries validation batch (e.g. early/unstable
training) triggered numpy RuntimeWarnings from .mean()/.std() on
empty arrays.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Covers the default-config 0-sentinel inheritance path (the exact bug
fixed by 969c5c6, previously untested since every router test in
test_router.py passes expert_hidden_dim/expert_n_blocks explicitly),
plus missing-key inheritance, full override, and partial override.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
out_dir (and thus the W&B run id, which is derived from out_dir.name)
was previously date-only, so two fresh runs on the same day with
identical hyperparams silently shared one W&B run history. Default
out_dir is now timestamped to the second. --resume without an explicit
--out now reuses the checkpoint's own parent directory instead of
recomputing a hyperparam-derived name, which both preserves the old
continue-in-place behavior and fixes a latent bug where a resumed run
with a changed hyperparam (e.g. --lr) would silently start writing to
a new directory.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Previously global_step always reset to 0, even on --resume. Since the
W&B run reattaches to the same run id on resume, logging with
step=global_step after a restart passed step values below what was
already recorded, silently dropping the resumed portion's metrics.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds Router.gate_stats (per-router gate entropy + per-expert utilization),
logged both per-batch (entropy only, train loop) and per-epoch (full
stats, over the whole val set) — the router-collapse failure mode from
the roadmap's rollout postmortem is now visible during training instead
of only after a full rollout+analysis run. Also splits WGAN critic/
generator grad norms instead of summing them, logs critic LR, n_sec head
accuracy, GPU peak memory + samples/sec, model parameter counts (in
wandb.config), and an is_best flag — all wired into both metrics.csv and
W&B.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in
DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed
run always got fixed 128/3-wide experts no matter what --hidden-dim/
--n-blocks was passed. They now default to 0 ("unset"), which
resolve_expert_dims() resolves by inheriting the model dims; an
explicit override still works and now warns when it diverges from
model.hidden_dim/n_blocks, since the checkpoint dir name won't
reflect it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant train --wandb now also logs loss/grad_norm/lr every N optimizer
steps (--wandb-log-every, default 50) so W&B shows within-epoch trends,
not just one point per epoch. Both share global_step as a single
monotonic step axis (wandb.Run.log requires step to never decrease
across calls), which also fixes global_step previously only advancing
in wgan mode.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant train --wandb logs the same per-epoch metrics already written to
metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`)
that nothing else depends on. A run's id is derived from the checkpoint
out_dir so --resume reattaches to the existing run instead of starting
a new one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The bind-mounted uv cache and the job workspace are on different
filesystems, so uv already falls back to copying installed files; this just
tells it to do so directly instead of logging a hardlink-failed warning
every job.
Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path
(/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow
env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step
right after setup-uv so it wins for the rest of the job.
setup-uv's default enable-cache: auto sets its own UV_CACHE_DIR (a tool-cache
tmp path) as a later step, clobbering the workflow-level UV_CACHE_DIR that
points at the bind-mounted persistent cache. Disable setup-uv's own cache
handling so our mount stays in effect.
The self-hosted act_runner doesn't support upload/download-artifact, so drop
that plumbing and instead run jobs in an explicit container with a persistent
host directory mounted at /uv-cache (UV_CACHE_DIR), backed by valid_volumes on
the runner. uv sync still runs per job but hits a warm local cache instead of
re-downloading/building packages every time.
Uploads the synced .venv as an artifact from a single setup job instead of
re-running uv sync (and re-downloading torch) in every job. Drops the
build/publish job in favor of a lighter job that, on a pushed tag, checks the
tag against the uv project version and — if they differ — bumps the version,
commits it to master, and recreates the tag on the new commit.
giant rollout now writes the checkpoint's complete model_config (incl.
the router sub-dict), the sibling config.toml's [train]/[meta] sections,
and every rollout CLI knob (weights, batch_size, escape_threshold,
n_events, device, seed) into the YAML sidecar instead of a hand-picked
subset. All of it flows through run_meta.json into each plot's own
metadata.yaml for later comparison, while the figure subtitle itself
shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router,
epoch, best_val_loss, steps/noise_dim) via new_figure's params option.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Records both identified contributors to the 2026-07-22 router divergence
(missing lambda_balance and the uniform-linspace center init) and that
the latter is now fixed, with next steps covering both for the retrain.
The 2026-07-22 rollout benchmark's router_gating diagnostic showed the
10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a
roughly uniform z-normalized energy distribution, leaving experts heavily
overlapping instead of partitioning the range. Add an optional
centers_init kwarg (backward compatible, defaults to the old linspace) and
have giant train estimate it from a reservoir sample of the real energy
column, collected during the existing normalizer-fitting pass.
The 10-expert EnergyRouter checkpoint (lambda_balance=0.0) diverges badly
from Geant4 on rollout (step granularity, secondary species, shower
shape), and the router_gating diagnostic shows the experts heavily
overlap rather than partitioning the pre-step energy domain. Recorded as
"needs retraining with a different router config" rather than an
abandoned direction. Full analysis in the knowledge base at
experiments/giant-router-energy-rollout-validation.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant analyze prep/submit previously defaulted the run directory to
next to the rollout parquet on /ceph. Default it instead to
<cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo
checkout (/work) — gitignored, --run-dir still overrides it.
derive_run_dir/prep gained a default_base param; library callers that
don't pass one keep the old parquet-relative fallback.
Checkpoints trained before commit 68fb99b (physical-property
conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond
normalizer, fit before build_cond_features grew the extra physical
columns. Any inference against such a checkpoint under current code
(predict/rollout/router_gating) crashed broadcasting a 15-wide
cond_cont against an 8-wide mean/std.
In "embedding" mode those physical columns are never read by
ConditionEncoder, so padding the missing entries with mean=0/std=1 is
a safe no-op. "physical" mode reads them directly, so a mismatch there
still raises instead of silently normalizing garbage.
The prior _COST_MODEL/_FIXED_OVERHEAD_S were fit only against local
synthetic benchmarks (up to 2M rows/side), which can't see docker
pull or real /ceph read latency and wildly overestimated real jobs
(~1200-1800s predicted vs 50-320s median actual, from condor_history
on production run 563f5ee3, --chunks 4, ~254M total rows).
Refit each spec's per-row rate through the origin against its median
real wall-clock time (not max, to avoid baking a few /ceph-contention
spikes into a rate that would then wrongly scale with dataset size),
and raised RUNTIME_SAFETY_MARGIN to compensate for that same
contention risk instead.
4096 MB was too tight: a resubmitted run held ~31 jobs spread evenly
across nearly every plot family and chunk index with "Docker job has
gone over memory limit of 4224 Mb", not one specific spec, so the
generic per-chunk data footprint needed more headroom.
mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current
numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9
not found). Switch the default to cverstege/alma9-gridjob, a modern
EL9-based image.
uv isn't installed on the HTCondor worker docker image, so `uv run`
fails there. giant is already an installed console script in the
repo's uv-synced .venv, so exec it directly instead. write_submit now
fails fast with a clear message if .venv/bin/giant is missing.
Each condor job's +RequestWalltime used to be one flat 3600s default
for every (plot, chunk), regardless of how much data it actually
streams over. `prep` now records each chunk's rollout+reference row
count, and `giant/analysis/runtime_estimate.py` turns that into a
per-job estimate: a per-spec (intercept, seconds/row) cost model fit
by `scripts/profile_analysis_costs.py` against synthetic mock data on
this machine, plus a fixed overhead placeholder (docker/uv/shared-fs
startup — unmeasurable here, no /ceph access) and a single
RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and
the submit description references it via $(walltime) instead of a
constant.
Add a second parallelism axis to giant analyze: each plot's data can now
be split into a configurable number of event_id-disjoint chunks, each
computed as its own HTCondor job, bounding per-job walltime and scan cost
on large rollout/reference files instead of one job re-scanning the
whole file per plot.
Every PlotSpec now splits into compute_partial (runs per (plot, chunk)
job against a chunk-filtered Bundle) and finalize (merges chunks -
elementwise sum for fixed-edge histograms/species shares, concatenate
-then-recompute for specs that derive edges or mean/std from the full
per-event/per-secondary array). Router diagnostics stay chunkable=False
and always run as a single job. giant analyze render now joins every
plot's chunk partials (merge_all) before rendering, transparently.
New: --chunks on `analyze prep`/`analyze submit`, --chunk on
`analyze compute-one`, and a new `analyze merge-one` command.
Rollout output and the reference file's ROOT-derived parquet disagree on
pdg's integer width (Int32 vs Int64), which only surfaced downstream as a
pl.concat SchemaError in build_context's pdg-count merge. Cast to a
canonical Int64 at the single scan entry point instead.
Adds a Compute environment section (laptop/desktop vs. shared portal
machines vs. condor workers) and corrects the roadmap: the WGAN-GP and
MoE routing-trunk tracks are actually implemented (untested and
under-testing respectively), not "not yet built" as previously stated.
Also notes the in-progress condor-gpu-train-rollout branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test-cuda.py was a one-off local CUDA check, not part of tests/ or scripts/.
docs/phase2_plan.md is superseded by the "Phase 2 (implemented)" section of
CLAUDE.md's roadmap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New "model" family in the gallery: router_gating (mean soft gate weight
vs. pre-step energy, showing the router's soft decision boundaries) and
router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share
by species / true physics process). Needs a live checkpoint's Router, so
it's a documented exception to the rest of the package's polars/numpy-only
contract; gracefully degrades to a placeholder for non-MoE checkpoints.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ty correctly flagged this as unsound: _CTX's inferred dict[str, int]
type doesn't rule out a "run_dir" key, which would silently bind to
prep's own run_dir: str | Path | None parameter instead of falling
through to **ctx_kwargs. Passing the context kwargs by name in a
small test helper removes the ambiguity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
plotstyle is now a real installed package (from the git.larsbogner.de
index) with its own type info, so the unresolved-import suppression
is no longer needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Swap the local/editable ETPlot `gallery[plotting]` dependency for
`plotstyle>=1.0.0`, now published to a package registry, so the repo
doesn't need a local ETPlot checkout to resolve.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`giant rollout` now records the checkpoint's architecture (mode,
hidden_dim, n_blocks, emb_dim, dropout, conditioning) plus training_epoch
and best_val_loss in its YAML sidecar, using data already loaded from the
checkpoint. condor.py carries those through run_meta, and render.py passes
them to plotstyle's new_figure(params=...) so every plot's subtitle shows
what produced it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`giant analyze prep` / `submit` now take the `giant rollout` YAML sidecar as
their only positional input instead of explicit --rollout/--reference/--out-dir.
The YAML's `output`/`dataset` keys name the rollout parquet and its seed file
(the reference truth), and the rest of the sidecar (checkpoint, geometry oracle,
cutoffs) flows into every plot's gallery metadata.
prep derives its own run directory next to the rollout parquet
(<...>/analysis_<id>/) holding shared.json, run_meta.json, reduced/, plots/.
compute-one and render now take just --run-dir / a run-dir argument and read the
resolved paths + metadata from run_meta.json, so the condor wrapper no longer
threads file paths. open_side scans a directory of reference shards via glob.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth
diagnostics) with a lean giant/analysis/ package that compares one
autoregressive `giant rollout` for a checkpoint against a held-out
miniCaloSim reference file, and generates publication-styled plots in
parallel on HTCondor.
Rollout output and a raw reference file share a world-frame physical
column subset under identical names, so the old ALR/local-frame decode
machinery is gone — everything is world-frame mm/MeV.
- sources.py: canonical LazyFrames, synthetic-termination-row filtering,
the secondary view (rollout generation>0 tracks vs reference sec_*_list).
- reduce.py: streaming primitives — a single hist1d group_by pass, per-event
scalars, edep-weighted depth/transverse profiles, species share, leakage.
- context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material
group sets once into shared.json, so each compute job is one pass, no range
scan (histogram efficiency).
- catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg,
material}, per-event totals, shower profiles, species/leakage, secondaries.
- render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata.
- condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit):
one job per plot, compute/render split (workers polars-only, no LaTeX).
Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover
the reduce primitives, catalog id uniqueness + compute, condor submit, and a
guarded render smoke test. Delete the two predict-diagnostics notebooks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ruff check, ruff format check, ty check, and pytest now run as four
independent jobs instead of one sequential lint job gating test —
faster wall-clock CI since none of these checks depend on each other.
build still waits on all four before bumping/publishing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Unquoted on: gets parsed as the YAML 1.1 boolean true instead of the
string "on", so Gitea couldn't find a trigger key and didn't register
the workflow at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three-stage workflow: lint/format/type-check (ruff, ty), pytest,
and (master pushes only) a version-bump + uv build + publish to the
Gitea package registry. Bump commit is tagged [skip ci] to avoid
retriggering the workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
plot_router_gating visualizes soft expert gate weights vs. a continuous
routing axis (e.g. pre-step energy), binned into equal-population
quantiles and stacked to show the router's soft decision boundaries.
Wired into rollout_validation.ipynb as a new notebook-only section
that loads a checkpoint's Router directly, since gate weights aren't
present in rollout/predict parquet output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars