The bind-mounted uv cache and the job workspace are on different
filesystems, so uv already falls back to copying installed files; this just
tells it to do so directly instead of logging a hardlink-failed warning
every job.
Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path
(/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow
env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step
right after setup-uv so it wins for the rest of the job.
setup-uv's default enable-cache: auto sets its own UV_CACHE_DIR (a tool-cache
tmp path) as a later step, clobbering the workflow-level UV_CACHE_DIR that
points at the bind-mounted persistent cache. Disable setup-uv's own cache
handling so our mount stays in effect.
The self-hosted act_runner doesn't support upload/download-artifact, so drop
that plumbing and instead run jobs in an explicit container with a persistent
host directory mounted at /uv-cache (UV_CACHE_DIR), backed by valid_volumes on
the runner. uv sync still runs per job but hits a warm local cache instead of
re-downloading/building packages every time.
Uploads the synced .venv as an artifact from a single setup job instead of
re-running uv sync (and re-downloading torch) in every job. Drops the
build/publish job in favor of a lighter job that, on a pushed tag, checks the
tag against the uv project version and — if they differ — bumps the version,
commits it to master, and recreates the tag on the new commit.
giant rollout now writes the checkpoint's complete model_config (incl.
the router sub-dict), the sibling config.toml's [train]/[meta] sections,
and every rollout CLI knob (weights, batch_size, escape_threshold,
n_events, device, seed) into the YAML sidecar instead of a hand-picked
subset. All of it flows through run_meta.json into each plot's own
metadata.yaml for later comparison, while the figure subtitle itself
shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router,
epoch, best_val_loss, steps/noise_dim) via new_figure's params option.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Records both identified contributors to the 2026-07-22 router divergence
(missing lambda_balance and the uniform-linspace center init) and that
the latter is now fixed, with next steps covering both for the retrain.
The 2026-07-22 rollout benchmark's router_gating diagnostic showed the
10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a
roughly uniform z-normalized energy distribution, leaving experts heavily
overlapping instead of partitioning the range. Add an optional
centers_init kwarg (backward compatible, defaults to the old linspace) and
have giant train estimate it from a reservoir sample of the real energy
column, collected during the existing normalizer-fitting pass.
The 10-expert EnergyRouter checkpoint (lambda_balance=0.0) diverges badly
from Geant4 on rollout (step granularity, secondary species, shower
shape), and the router_gating diagnostic shows the experts heavily
overlap rather than partitioning the pre-step energy domain. Recorded as
"needs retraining with a different router config" rather than an
abandoned direction. Full analysis in the knowledge base at
experiments/giant-router-energy-rollout-validation.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant analyze prep/submit previously defaulted the run directory to
next to the rollout parquet on /ceph. Default it instead to
<cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo
checkout (/work) — gitignored, --run-dir still overrides it.
derive_run_dir/prep gained a default_base param; library callers that
don't pass one keep the old parquet-relative fallback.
Checkpoints trained before commit 68fb99b (physical-property
conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond
normalizer, fit before build_cond_features grew the extra physical
columns. Any inference against such a checkpoint under current code
(predict/rollout/router_gating) crashed broadcasting a 15-wide
cond_cont against an 8-wide mean/std.
In "embedding" mode those physical columns are never read by
ConditionEncoder, so padding the missing entries with mean=0/std=1 is
a safe no-op. "physical" mode reads them directly, so a mismatch there
still raises instead of silently normalizing garbage.
The prior _COST_MODEL/_FIXED_OVERHEAD_S were fit only against local
synthetic benchmarks (up to 2M rows/side), which can't see docker
pull or real /ceph read latency and wildly overestimated real jobs
(~1200-1800s predicted vs 50-320s median actual, from condor_history
on production run 563f5ee3, --chunks 4, ~254M total rows).
Refit each spec's per-row rate through the origin against its median
real wall-clock time (not max, to avoid baking a few /ceph-contention
spikes into a rate that would then wrongly scale with dataset size),
and raised RUNTIME_SAFETY_MARGIN to compensate for that same
contention risk instead.
4096 MB was too tight: a resubmitted run held ~31 jobs spread evenly
across nearly every plot family and chunk index with "Docker job has
gone over memory limit of 4224 Mb", not one specific spec, so the
generic per-chunk data footprint needed more headroom.
mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current
numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9
not found). Switch the default to cverstege/alma9-gridjob, a modern
EL9-based image.
uv isn't installed on the HTCondor worker docker image, so `uv run`
fails there. giant is already an installed console script in the
repo's uv-synced .venv, so exec it directly instead. write_submit now
fails fast with a clear message if .venv/bin/giant is missing.
Each condor job's +RequestWalltime used to be one flat 3600s default
for every (plot, chunk), regardless of how much data it actually
streams over. `prep` now records each chunk's rollout+reference row
count, and `giant/analysis/runtime_estimate.py` turns that into a
per-job estimate: a per-spec (intercept, seconds/row) cost model fit
by `scripts/profile_analysis_costs.py` against synthetic mock data on
this machine, plus a fixed overhead placeholder (docker/uv/shared-fs
startup — unmeasurable here, no /ceph access) and a single
RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and
the submit description references it via $(walltime) instead of a
constant.
Add a second parallelism axis to giant analyze: each plot's data can now
be split into a configurable number of event_id-disjoint chunks, each
computed as its own HTCondor job, bounding per-job walltime and scan cost
on large rollout/reference files instead of one job re-scanning the
whole file per plot.
Every PlotSpec now splits into compute_partial (runs per (plot, chunk)
job against a chunk-filtered Bundle) and finalize (merges chunks -
elementwise sum for fixed-edge histograms/species shares, concatenate
-then-recompute for specs that derive edges or mean/std from the full
per-event/per-secondary array). Router diagnostics stay chunkable=False
and always run as a single job. giant analyze render now joins every
plot's chunk partials (merge_all) before rendering, transparently.
New: --chunks on `analyze prep`/`analyze submit`, --chunk on
`analyze compute-one`, and a new `analyze merge-one` command.
Rollout output and the reference file's ROOT-derived parquet disagree on
pdg's integer width (Int32 vs Int64), which only surfaced downstream as a
pl.concat SchemaError in build_context's pdg-count merge. Cast to a
canonical Int64 at the single scan entry point instead.
Adds a Compute environment section (laptop/desktop vs. shared portal
machines vs. condor workers) and corrects the roadmap: the WGAN-GP and
MoE routing-trunk tracks are actually implemented (untested and
under-testing respectively), not "not yet built" as previously stated.
Also notes the in-progress condor-gpu-train-rollout branch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
test-cuda.py was a one-off local CUDA check, not part of tests/ or scripts/.
docs/phase2_plan.md is superseded by the "Phase 2 (implemented)" section of
CLAUDE.md's roadmap.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New "model" family in the gallery: router_gating (mean soft gate weight
vs. pre-step energy, showing the router's soft decision boundaries) and
router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share
by species / true physics process). Needs a live checkpoint's Router, so
it's a documented exception to the rest of the package's polars/numpy-only
contract; gracefully degrades to a placeholder for non-MoE checkpoints.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ty correctly flagged this as unsound: _CTX's inferred dict[str, int]
type doesn't rule out a "run_dir" key, which would silently bind to
prep's own run_dir: str | Path | None parameter instead of falling
through to **ctx_kwargs. Passing the context kwargs by name in a
small test helper removes the ambiguity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
plotstyle is now a real installed package (from the git.larsbogner.de
index) with its own type info, so the unresolved-import suppression
is no longer needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Swap the local/editable ETPlot `gallery[plotting]` dependency for
`plotstyle>=1.0.0`, now published to a package registry, so the repo
doesn't need a local ETPlot checkout to resolve.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`giant rollout` now records the checkpoint's architecture (mode,
hidden_dim, n_blocks, emb_dim, dropout, conditioning) plus training_epoch
and best_val_loss in its YAML sidecar, using data already loaded from the
checkpoint. condor.py carries those through run_meta, and render.py passes
them to plotstyle's new_figure(params=...) so every plot's subtitle shows
what produced it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`giant analyze prep` / `submit` now take the `giant rollout` YAML sidecar as
their only positional input instead of explicit --rollout/--reference/--out-dir.
The YAML's `output`/`dataset` keys name the rollout parquet and its seed file
(the reference truth), and the rest of the sidecar (checkpoint, geometry oracle,
cutoffs) flows into every plot's gallery metadata.
prep derives its own run directory next to the rollout parquet
(<...>/analysis_<id>/) holding shared.json, run_meta.json, reduced/, plots/.
compute-one and render now take just --run-dir / a run-dir argument and read the
resolved paths + metadata from run_meta.json, so the condor wrapper no longer
threads file paths. open_side scans a directory of reference shards via glob.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth
diagnostics) with a lean giant/analysis/ package that compares one
autoregressive `giant rollout` for a checkpoint against a held-out
miniCaloSim reference file, and generates publication-styled plots in
parallel on HTCondor.
Rollout output and a raw reference file share a world-frame physical
column subset under identical names, so the old ALR/local-frame decode
machinery is gone — everything is world-frame mm/MeV.
- sources.py: canonical LazyFrames, synthetic-termination-row filtering,
the secondary view (rollout generation>0 tracks vs reference sec_*_list).
- reduce.py: streaming primitives — a single hist1d group_by pass, per-event
scalars, edep-weighted depth/transverse profiles, species share, leakage.
- context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material
group sets once into shared.json, so each compute job is one pass, no range
scan (histogram efficiency).
- catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg,
material}, per-event totals, shower profiles, species/leakage, secondaries.
- render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata.
- condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit):
one job per plot, compute/render split (workers polars-only, no LaTeX).
Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover
the reduce primitives, catalog id uniqueness + compute, condor submit, and a
guarded render smoke test. Delete the two predict-diagnostics notebooks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ruff check, ruff format check, ty check, and pytest now run as four
independent jobs instead of one sequential lint job gating test —
faster wall-clock CI since none of these checks depend on each other.
build still waits on all four before bumping/publishing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Unquoted on: gets parsed as the YAML 1.1 boolean true instead of the
string "on", so Gitea couldn't find a trigger key and didn't register
the workflow at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three-stage workflow: lint/format/type-check (ruff, ty), pytest,
and (master pushes only) a version-bump + uv build + publish to the
Gitea package registry. Bump commit is tagged [skip ci] to avoid
retriggering the workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
plot_router_gating visualizes soft expert gate weights vs. a continuous
routing axis (e.g. pre-step energy), binned into equal-population
quantiles and stacked to show the router's soft decision boundaries.
Wired into rollout_validation.ipynb as a new notebook-only section
that loads a checkpoint's Router directly, since gate weights aren't
present in rollout/predict parquet output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds --mode wgan alongside flow/ddpm: both stages get a WGAN-GP
generator/critic pair (giant.model.wgan) instead of flow matching, so
inference is a single forward pass per stage rather than a 10-step ODE
integration — the fast-eval architecture noted in the roadmap.
predict/rollout auto-detect the mode from the checkpoint's model_config.
Best-checkpoint selection for wgan uses marginal-KL against the EMA
generators every epoch, since a critic loss isn't a monotone quality
signal. --router is not supported together with --mode wgan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- make_seed_frontier only resolves particle mass/charge in "physical"
mode, so "embedding"-mode rollouts no longer crash on a seed PDG code
giant.particles can't resolve (the TERM_UNKNOWN_PDG gate now handles it).
- nearest_known_pdg skips unresolvable candidate PDG codes instead of
raising and killing the whole rollout/predict run.
- predict/rollout fail with a clear message when a checkpoint predates
the sec_phys normalizer, instead of a bare KeyError.
- validate_marginals' phys_kl degrades to NaN (matching the
energy_fraction_kl pattern) instead of crashing when a validated batch
has zero secondaries on either side.
- Correct CLAUDE.md's stale claim that the materials table is unfilled.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds model.conditioning = "physical" | "embedding": physical mode routes
particle mass/charge and material Z_eff/A_eff/density/X0/lambda_int through
small MLPs to replace the learned PDG/material embedding tables, so the
surrogate generalizes to PDG codes/materials outside the training vocab
instead of memorizing it. "embedding" stays available as the comparison
baseline (old checkpoints without the key default to it).
Stage 2 now regresses a secondary's mass/charge directly against a fixed
physics-derived target instead of a learned/snapped embedding, and uses no
snapping at inference — the model's raw predicted (mass, charge) is the
secondary's physical identity, including for its own further rollout steps.
A separate reporting-only nearest-known-PDG lookup (never fed back into the
model) populates output pdg columns / the embedding-mode rollout fallback.
giant/materials.py's table is populated with Geant4's own built-in NIST
constants (Z_eff, A_eff, density, X0, lambda_int), extracted directly from
the Geant4 11.4.1 build vendored in minicalosim via G4NistManager rather
than hand-typed literature values. G4_LYSO is left unfilled: confirmed (both
by runtime lookup and by searching minicalosim's history) that it's never
actually a constructed Geant4 material there, only documentation/UI color-map
text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merged in every non-analysis change from the MoE-prototype branch (routing,
training, data pipeline, streaming rollout output), keeping this branch's
lean streaming giant/analysis.py and rebuilding the rollout-vs-truth feature
natively on it instead of resurrecting the old numpy SampleCollection path.
- Add RolloutVsTruth, accepted anywhere Tier 1-3 functions take a predict-parquet
source: decodes a giant rollout file and a held-out truth file into
RAW_TARGET_NAMES space via a polars port of the forward local-frame rotation,
fully streaming (no SampleCollection, no eager materialization).
- Add compute_rollout_vs_truth_observables_pl for Tier 4, reusing
EventObservables (now backed by independent real_table/gen_table to support
unequal rollout/truth event counts) so every existing shower-observable plot
function works unchanged for both one-step and full-rollout comparisons.
- Update analysis/rollout_validation.ipynb to the new API and CLAUDE.md's
architecture description; add test coverage for the new source type.
- Fix a pre-existing return-type mismatch in giant.rollout.rollout() (found by
`ty check`): the on_chunk summary-dict branch didn't match the declared
dict[str, np.ndarray] return type, now expressed as a RolloutSummary TypedDict.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives flow-matching sampling a cleaner EMA shadow copy to draw from (--ema-decay,
--weights raw|ema in predict/rollout), fixes the LR warmup/cosine schedule stepping
once per epoch even when an epoch is tens of thousands of steps, and caps the
per-epoch val-loss pass (--max-val-batches) so large val sets don't dominate epoch time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Training runs the full soft mixture (every expert over the whole batch),
so routed activation memory scales with the expert count; the old estimate
used one expert's dims and would overshoot free VRAM by a factor of
n_experts. Fold the expert count into n_blocks for the training path
(inference's top-1 dispatch still just partitions the batch, so one
expert's dims bound it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route on several independent axes at once (e.g. energy x pdg), each with
its own expert count and hyperparameters. The joint gate is the outer
product of per-axis softmax gates, so it stays a partition of unity and
top1/balance_loss factor per-axis. Config uses flat axis{i}_{field} keys
in model.router (TOML/CLI friendly), also settable via repeatable
--router-axis flags.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_Recorder previously accumulated every generated step across all events/
tracks/steps in Python lists, materialised once at the end and written
via a single pq.write_table — memory scaled with n_events * max_steps *
avg_tracks_per_event. rollout() now takes an optional on_chunk callback
that streams each non-empty batch immediately (fixed per-key dtypes via
_RECORD_DTYPES keep every chunk's table schema identical, which
pq.ParquetWriter requires across writes); giant rollout wires this to an
incrementally-written ParquetWriter, mirroring the row-group streaming
giant predict already does on its input side. Without on_chunk, rollout()
keeps its old buffered return for existing callers/tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Routes on the pre-step PDG code, which — unlike ProcessRouter's process
label — is already known at gate time (a conditioning input), so no
supervision is needed and classify_loss falls back to the zero default.
Generalizes EnergyRouter's soft-turn-on-then-Voronoi trick from a 1-D
distance to a small learned PDG embedding space: its own embedding table
maps each PDG code to a point, and n_experts learnable centers partition
that space.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars