shower_containment_depth_90/95's title contains a literal "%" (e.g.
"...(90% of deposited energy)"), which usetex reads as a comment marker
and aborts LaTeX compilation. Since render_all processes reduced JSON
files in sorted filename order, this killed every plot id sorting after
these two in the same run.
Escape title/xlabel once, centrally, in render()'s dispatch (the one
place every renderer kind draws them from before handing off to
plotstyle/matplotlib) rather than at each catalog.py call site, so any
future catalog title with a %, &, #, etc. is covered automatically.
_plot_metadata keeps using the unescaped Reduced for the gallery YAML,
since that's not LaTeX.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant analyze compares N rollout YAMLs against one shared reference file
(all must name the same dataset, checked up front) instead of exactly one
rollout vs one reference, rendering each rollout as its own colored series
against a single reference line/panel. Series names come from a repeated
--label flag, else the YAML stem, else "rollout" for a single YAML — a
single-rollout run keeps rendering identically to before this change.
Bundle now holds a name-keyed dict of rollout sides instead of one fixed
pair, every catalog compute_partial/finalize builds a Reduced.payload
keyed the same way ("series": {name: ...}, "reference": ... as the one
distinguished non-rollout entry), and every renderer draws N series (or
N panels, for the two heatmap-shaped specs and the router/type-embedding
diagnostics, which are inherently one-matrix/one-checkpoint per rollout)
against the reference's fixed dashed-ink style.
CliRunner stores an uncaught exception in result.exception, not
result.output, so the skip condition never matched and the test
failed outright on CI machines without LaTeX installed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Picks 4 of the 7 catalog additions the issue proposed (the smaller-lift
ones; 2D joint plots, PIT calibration, and the throughput/accuracy scatter
are left for follow-up issues):
- marginal_distance_summary: a var x grouping-axis KS-statistic heatmap,
reusing the existing marginal hist1d compute and just adding a finalize —
a single at-a-glance regression scorecard instead of N overlay plots.
- n_sec_confusion: predicted (rollout) vs true (reference) secondary count
per event, paired by event_id since a rollout is seeded from the same
events as its reference file. Needed a new zero-filling primitive
(reduce.sec_count_by_event) since a plain group_by over secondary rows
silently drops zero-secondary events.
- shower_containment_depth_{90,95}: per-event depth containing 90%/95% of
deposited energy, derived from the same per-event depth-bin matrix the
longitudinal profile already computes.
- router_specialization: max gate weight vs energy per side, summarizing
router_gating's full stacked area into the one trend line the roadmap's
MoE writeup describes (the ~60-65% ceiling), to make a future
lambda_balance>0 retrain's effect on specialization checkable at a glance.
Both new heatmap-shaped plots (distance summary, confusion matrix) share one
new "heatmap" Reduced kind/renderer rather than two near-identical ones.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MetricsCollector writes one row per epoch to <run_dir>/metrics.csv, but
nothing read or plotted it. giant/training/plots.py reads the CSV header
dynamically (the column set varies by run: flow/ddpm vs wgan, routed vs
not) and renders loss/lr/accuracy/grad-norm/router/wgan-balance/throughput
plots with the same plotstyle conventions giant/analysis/render.py uses,
skipping any figure whose columns aren't present for a given run.
Wired up as `giant analyze metrics <run_dir>`, writing PDFs into the same
gitignored analysis_runs/ directory `analyze prep`/`submit` already use
(derive_metrics_dir mirrors derive_run_dir) rather than into the training
run directory itself.
CriticModel was the one stage-shaped class left out of the trunk-registry
(gitea #33), block-conditioning-registry (gitea #34), and StageModel-base
(gitea #39) refactors: it hand-rolled a plain ResBlock stack, so a
routed/FiLM/AdaLN trunk was available to every generative stage model except
the critic competing against them under WGAN-GP.
CriticModel now subclasses StageModel (reusing its cond_enc construction, and
a stage-2 context-fusion helper factored out of Stage2OneShot onto the base)
and builds its body via build_trunk (output width 1) instead of a bespoke
ResBlock loop, so trunk.type/trunk.block_conditioning now affect the critic
too. Each stage's critic inherits its own generator's trunk config rather
than a new critic_trunk config key, mirroring the existing
critic_hidden_dim/critic_n_res_blocks "0 = inherit from generator" pattern.
Router mixing (MoE) for the critic stays out of scope. Since CriticModel is
training-only and never persisted for inference, and WGAN-GP is still
unbenchmarked, its state_dict shape has no back-compat burden.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Turns "is this component earning its parameters?" into a one-line
config flip for each of the three pluggable network components:
- router.type = "none" (NoneRouter, giant/model/routers.py): still
builds n_experts expert trunks via RoutedTrunk, but replaces the
learned gate with a uniform 1/n_experts weight for every row — no
centers/embeddings/classifier. Distinct from router.enabled=false
(which drops routing/mixing entirely): this isolates whether the
*learned routing signal* specifically is earning its parameters,
holding expert count fixed.
- stage2_model.autoregressive.history = "none" (NoHistory,
giant/model/history.py): ignores feat/has_prev entirely and always
returns zeros, ablating whether the AR decoder's history
conditioning earns its parameters. Already validated for free by
gitea #35's generic HISTORY_REGISTRY membership check.
- trunk.type = "linear" (LinearTrunk, giant/model/trunks.py): a bare
nn.Linear(in_dim + cond_dim, out_dim) body, no ResBlock stack. Per
gitea #33's design, this composes for free with router.enabled=true
("mixture of trivial linear experts").
Both blocking issues (#33 trunk registry, #35 pluggable history
encoder) are closed, so this was unblocked.
The prior e2e run (task 1348) failed on the bump-version job's push step
with a 403 Forbidden — CI_TOKEN lacked write access. Note this on the
checkout step so the requirement isn't lost, now that the token has been
rescoped. Trivial commit to re-open a merge request and re-run the job
end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version bumps and release tags were entirely manual; the only CI automation
was sync-version-on-tag, which corrects pyproject.toml if a hand-pushed tag
drifted. This flips that: a new bump-version job (needs the four existing
checks, gated to actual merge commits on master via HEAD^@'s parent count so
direct/squash/rebase pushes are untouched) uses bump-my-version to auto-bump
the patch version when a merged branch didn't already bump it itself, then
generates a changelog entry with git-cliff and pushes a matching vX.Y.Z tag.
git-cliff's cliff.toml is tuned to this repo's plain imperative commit style
(no feat:/fix: prefixes): commits are grouped Added/Fixed/Removed/Changed by
leading verb, "(gitea #N)" is linkified, and merge/[skip ci] commits are
dropped. Per user decision during planning: the changelog generator folds in
@lars's comment on the issue (asking to fold in changelog generation rather
than deferring it), and CHANGELOG.md starts fresh with no backfill of
v0.2.0-v0.3.3.
sync-version-on-tag is left untouched as the safety net for hand-tagging.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v0.3.0 pivot exists because the 2026-08-03 WGAN rollout benchmark
produced zero photon secondaries and ~4M hallucinated antineutrinos —
even with a correctly-sized top-N species vocabulary (gitea #29), plain
cross-entropy over a class distribution spanning orders of magnitude
still under-predicts rare-but-physical species.
stage2_model.particle_type.class_weighting = "none" | "inverse_freq"
(default "none", fully back-compat) weights the stage-2 type head's CE
loss (FlowDDPMStageTrainer._type_loss) by inverse class frequency,
normalized to mean 1 so switching it on doesn't rescale the type loss
against particle_type.lambda / the generator loss it's summed with.
The per-class counts the weighting needs don't already exist despite the
issue's premise: _topn_plus_other_map (giant/data/loader.py) previously
kept counts only for keys folded into "other", dropping the kept classes'
counts on the floor. TopNMap now carries class_counts (index -> count),
round-tripped through the setup-cache sidecar (format version bumped
3->4, since existing sidecars have none) and through checkpoints
(tolerantly — a pre-#44 checkpoint decodes to {}, since only training-time
loss weighting reads it, not inference).
Decisions made during planning (with the user): dropped "effective_num"
from the issue's proposed three-way enum (no beta hyperparameter to
design around) — final domain is "none" | "inverse_freq". Weights are
mean-1-normalized. validate_config rejects class_weighting != "none"
combined with particle_type.target != "onehot" or
stage2_model.generator == "wgan" (both have no class CE to weight),
following the #28/#30 dead-key-must-not-go-silent convention. Branch
fix/issue-44 off master.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant/analysis/sources.py's open_side scanned a reference directory of
parquet shards with a bare glob and never offset event_id across them.
Each shard is a separate Geant4 job whose own event_id numbering restarts
from 0, so events from different shards collided on the same event_id,
corrupting every downstream per-event grouping and the event_id % n_chunks
condor chunking — the same root cause already fixed on the training/rollout
side via giant/data/loader.py's per-file event_id_offset.
open_side's reference branch now uses find_parquet_files (the same
deterministically ordered file lister giant rollout's _seed_from_data uses)
and offsets each shard's event_id via a join on polars' include_file_paths,
so both sides of a comparison agree on what an event_id means. Two
incidental behaviour changes come along for free: .manifest references now
work (they crashed before), and the directory glob narrows from recursive
**/*.parquet to top-level *.parquet, matching the file list rollout itself
used to assign offsets — a deliberate choice, since a differing file list
would make the two sides' offsets disagree again in a subtler way.
No overflow guard on the per-shard offset stride (unlike loader's
_offset_event_id): checking it here would cost an eager event_id-column
read per shard in every condor compute job, and giant rollout already runs
that check over the same file list when producing the seed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant/ had no autocast/GradScaler/torch.compile anywhere despite the
project's ~10x-native-Geant4 eval-budget target. This adds bf16 mixed
precision to the training step (both FlowDDPMStageTrainer and
WGANStageTrainer) via a new train.precision config key ("fp32" default,
"bf16" opt-in) and giant.training.amp.resolve_autocast.
torch.compile is a separate, much larger surface (data-dependent routed
dispatch, the autoregressive sampler's per-token control flow, arbitrary
rollout batch sizes) and is left for a follow-up issue, per discussion.
Scope decisions made during planning:
- fp32 + bf16 only, no fp16/GradScaler. fp16 breaks two things in this
codebase: routers.py's three 1e-8 epsilons sit below fp16's ~6e-8
subnormal floor, and gradient_penalty's grad norm overflows fp16's
range at ordinary early-WGAN-GP gradient magnitudes. Every training
GPU in the fleet (A100/L40S/H200/RTX 4070) has native bf16; only
pre-Ampere V100s would need fp16.
- resolve_autocast raises loudly if bf16 is requested on hardware that
can't do it, rather than silently falling back to fp32.
- Autocast wraps the training step only; val_loss (and the
best-checkpoint selection it drives) stays fp32 so it's comparable
across every run recorded so far.
- _route_forward's mixture accumulator (giant/model/trunks.py) was a
hard-fp32 torch.zeros with no dtype, so under autocast a RoutedTrunk
silently returned a different output dtype than an unrouted
ExpertTrunk purely because router.enabled was set. Fixed to match the
experts' own dtype; the gate weights (forced fp32 for their own
numerical stability) are cast down before combining, so the
mixture's numerics stay solid without reintroducing the dtype split.
- Added explicit fp32 guards (autocast(enabled=False)) around spots
that are correct in fp32 but degrade quietly rather than crash in
bf16: the router's balance/entropy losses and gate softmax, the
stage-2 stick-breaking cumprod, and gradient_penalty's
double-backward + grad norm.
Benchmarked on the local RTX 4070 against configs/baseline.toml's
hyperparams (hidden_dim 512/6 blocks, bs 4096) on a synthetic dataset:
bf16 gave 1.05-1.35x training throughput and 18-33% lower peak GPU
memory across one-shot/routed/autoregressive stage-2 configs, with the
autoregressive path (the dominant cost per baseline.toml) benefiting
most on both axes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
stage{1,2}_model.active = false already trains one stage alone, but the
checkpoint it writes holds only that stage, so giant rollout refuses it --
the "retrain stage 2 alone against a fixed, known-good stage 1" experiment
the 2026-08-03 species failure calls for wasn't runnable end to end.
Adds stage{1,2}_model.init_from (a checkpoint .pt to load this stage's
weights from before training) and .freeze (never update them), symmetric
across both stages. Both stages stay active = true, so both get built and
both land in the output checkpoint -- the frozen stage is merely
initialized from disk instead of from scratch.
Decisions made during planning:
- Soft freeze: forward/backward still run every batch (loss/grad_norm stay
meaningful, no autograd special-casing), only optimizer.step() (and, for
the frozen stage, lr_sched.step()/EMA update) is skipped -- weights are
byte-identical for the whole run. This is StageTrainer._step_optimizer,
shared by the non-adversarial path and both halves (generator + critic)
of the WGAN path, so a frozen WGAN stage's critic freezes too.
- validate_config requires init_from whenever freeze = true, unless the run
is a --resume (a resumed frozen stage's weights come from the resume
checkpoint instead) -- freezing a randomly-initialized model is almost
certainly a mistake.
- CLI flags on both `giant train` and `giant new-run`
(--stage{1,2}-init-from/--stage{1,2}-freeze), matching every other
per-stage model knob's existing treatment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 2 was trained on ground-truth stage-1 outcomes but deployed on
sampled ones, and in a rollout that gap compounds over every step of
every track — the same train/inference gap teacher_forcing="scheduled"
already closes within stage 2, just never applied at the stage
boundary. "sampled" was declared in the schema but rejected loudly by
validate_config as unimplemented; this lands the real implementation.
Mirrors the existing scheduled-sampling precedent rather than a hard
switch: new stage2_model.ctx_p_start/ctx_p_end (defaults 1.0 -> 0.0)
linearly ramp P(condition on ground truth) from epoch 0 to the final
epoch, so stage 2 doesn't chase a wildly moving stage-1 target early in
training. Per the plan discussed with the user: the sample is drawn
from stage 1's sampling_model() (EMA weights when present, matching
what inference actually deploys), mixed per example via a Bernoulli
draw (never blended within a row), and validation always uses the
ground truth regardless of the schedule. Fixes a latent bug the same
pattern would otherwise have hit: every sampler in giant/sample.py
flips its model to .eval() with no restore, so sampling from the raw
(non-EMA) stage-1 model mid-step now explicitly restores its .training
flag afterward to avoid silently corrupting stage 1's own training mode
for the rest of the epoch.
validate_config now enforces stage1_context in {"truth", "sampled"},
requires both stages active for "sampled" (nothing to sample from
otherwise), range-checks ctx_p_start/ctx_p_end, and rejects the
ctx_p_start = ctx_p_end = 1.0 configuration as an unadvertised no-op
identical to "truth".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
giant model summary --config config.toml builds the resolved Stage1/Stage2
graph from a config with no dataset attached (pdg_vocab/mat_vocab are
supplied as placeholders via --pdg-vocab/--mat-vocab, since the real
training vocab is dataset-derived) and prints per-module parameter counts,
trunk in/out widths, which heads exist, and which
conditioning/stage1_model/stage2_model config keys actually shaped the
build.
The consumed-keys half uses differential probing rather than static
identifier matching: build once for a fingerprint (submodule presence,
every parameter's/buffer's shape+dtype, every plain scalar attribute a
module stores on itself), then perturb one leaf at a time, rebuild, and
compare. A changed fingerprint (or a raise) means the key is consumed; no
change means it's inert *under this particular config* -- e.g. any
stage1_model.router.* key when router.enabled=false. A curated
_NOT_BUILD_TIME table separates keys legitimately owned by the
trainer/sampler/rollout (loss weights, WGAN-GP hyperparameters,
teacher-forcing schedules) from genuinely-inert ones, verified against
those call sites. A few config keys branch on equality against one specific
string literal (n_sec.owner=="stage1", n_sec.mode=="stop_token",
particle_type.target=="physical"); a single generic sentinel probe missed
all three since the config's current value and the sentinel landed in the
same branch, so those three leaves get their real alternative value tried
too (_STRING_ALTERNATIVES).
giant.config.leaf_paths is promoted out of
tests/test_config_consumed_keys.py (previously a private test-local
duplicate) so both audits -- the static per-identifier one and this new
runtime per-config one -- walk the exact same DEFAULT_CONFIG tree.
ExpertTrunk/RoutedTrunk now also expose in_dim (out_dim already existed),
needed to report trunk widths generically.
Decisions made during planning: --pdg-vocab/--mat-vocab default to 300 and
len(MATERIAL_PROPERTIES); the consumed-keys report is scoped to
conditioning/stage1_model/stage2_model only (train/meta are out of scope
for a model-only build); the module tree prints every submodule at any
depth.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 2's autoregressive decoder still predicted multiplicity the v0.2 way:
a one-shot n_sec_head classifier over conditioning alone, run before any
secondary token existed, with the AR loop then always executing k_max slots
and discarding the tail. This adds a real per-slot EOS mechanism instead:
- Stage2Autoregressive gains a stop_head (build_stop_head=True) that predicts
P(n_sec == k | prefix) at each slot, mutually exclusive with n_sec_head
(n_sec.mode = "stop_token" builds no n_sec_head at all).
- sample_secondaries_ar accepts n_sec_pred=None to drive generation off the
stop head instead of a pre-resolved count: each row stops the first slot
its stop logit fires (stage2_model.n_sec.stop_sampling = "greedy" — the
default, threshold at 0 — or "sample", a Bernoulli draw), and the whole
batch loop breaks once every row has stopped, so cost scales with the
realized n_sec instead of a fixed k_max. Passing n_sec_pred explicitly
(the scheduled-sampling self-sample path) is unchanged.
- resolve_n_sec returns None for a stop-token decoder instead of raising;
rollout.py/cli.py/validate.py now derive the realized count from
sample_stage2's returned sec_valid (sec_valid.sum(-1)) after sampling,
rather than resolving it up front — a no-op reordering under every other
n_sec.mode, where sec_valid was already built from n_sec_pred.
- Training: _stop_target_and_mask (giant/training/stage2_inputs.py) builds
the per-slot target/mask (one slot wider than the existing token-content
sec_mask, since the stop slot itself needs supervision) and
StageTrainer._stop_loss trains it with masked BCE, gated on stop_head
exactly like _n_sec_loss gates on n_sec_head. Wired into both the
flow/ddpm trainer and the WGAN trainer (whose skip_g_step now also checks
stop_head), weighted by the existing stage2_model.n_sec.lambda — the stop
head replaces n_sec_head under this mode, so no new weight key.
- validate_config now accepts stop_token (requires decoder="autoregressive"
and n_sec.owner="stage2") instead of always rejecting it.
Decisions made during planning: stop_sampling defaults to "greedy" for
deterministic rollouts; the stop head reuses stage2_model.heads.n_sec's
HeadConfig shape and stage2_model.n_sec.lambda's weight rather than adding
new config keys, since the two heads never coexist.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_bin_expr in giant/analysis/reduce.py clipped the bin index to
[0, nbins-1] only after casting it to Int32, so the clip never got a
chance to run: a rollout step_length of 1.0725e10 mm against fixed
edges [2.9e-5, 94.04] with 50 bins produces a raw index of ~5.7e9,
which overflows i32 and fails the strict cast, killing the whole
compute-one job. Same failure mode for +/-inf.
Clamp in f64 first, then cast to Int32. NaN has no edge to clamp to,
so it maps to null and is dropped in the two callers (hist1d,
profile_partial) — matching what np.histogram does with NaN, and what
profile_partial needs anyway since a null bin index would break its
np.add.at.
This reimplements commit 313373c, which fixed the same bug but landed
on a branch (fix/rollout-negative-secondary-mass) that forked off a
stale master and was never merged; reduce.py has since diverged enough
that the original diff no longer applies cleanly.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
gitea-actions
lars