giant predict/rollout rebuilt models straight from ckpt["model_config"] with
no way to change sampling-only keys (e.g. stage2_model.n_sec.stop_sampling)
without retraining. Adds config_overrides to load_for_inference, validated
against giant.config.INFERENCE_OVERRIDES so a typo or shape-bearing key
raises CheckpointCompatibilityError up front instead of an opaque
load_state_dict mismatch. Wired as a repeatable --set dotted.path=value on
both CLI commands, recorded in the rollout YAML sidecar, and surfaced in
`giant model summary`'s output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Taking argmax over the n_sec classifier logits collapses secondary
multiplicity onto its conditional mode at fixed pre-step conditioning,
under-dispersing n_sec in rollouts and biasing low wherever the true
conditional count distribution is right-skewed (typical for
multiplicity).
Generalizes stage2_model.n_sec.stop_sampling (previously stop_token-only)
into stage2_model.n_sec.sampling, covering both "head" (greedy: argmax;
sample: categorical draw via torch.multinomial) and "stop_token" (unchanged:
greedy threshold / Bernoulli draw) modes. stop_sampling is kept as a
deprecated alias in NSecConfig.from_dict and migrate_config, since it
appears in existing checkpoints' model_config. Default stays "greedy" so
existing runs/checkpoints are unaffected.
Replace the event-level n_sec confusion matrix with two step-resolved
secondary-multiplicity comparisons:
- sec_count_per_step: overlay histogram of how many secondaries a single
step emits, rollout series vs reference.
- sec_count_per_step_by_species: heatmap of per-step multiplicity of one
species (zero row included) against species, drawn as one panel per
rollout plus a reference panel, raw counts on a log color scale.
Both are backed by a new sources.secondaries_by_step view, which tags each
secondary with its emitting step — (event_id, parent_id, birth position)
on the rollout side, the row index on the reference side — so neither plot
needs a join against the step frame. Steps that emitted nothing are
recovered by subtraction from the chunk's step count, keeping both specs
sum-mergeable across condor chunks.
The rollout multiplicity is derived from the actual secondary birth rows
rather than the n_sec_pred column, which records the predicted count
before the per-event max-tracks cap.
_render_heatmap gained reference-panel and log-color support;
marginal_distance_summary sets neither key and is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLAUDE.md still described the pre-v0.3.0 codebase: the Stage-2
autoregressive redesign as "designed, not implemented", a monolithic
network.py, a single global model.conditioning switch, and WGAN as
"implemented, not yet tested".
- Architecture rewritten around the actual giant/model split
(layers/encoders/trunks/routers/history/objectives/models/builders/
_legacy/summary; network.py is now a re-export shim), plus
cond_layout.py, checkpoint_io.py, _migration.py, data/setup_cache.py
and giant/training/.
- Conditioning documented per axis (conditioning.particle /
conditioning.material, each physical|embedding|onehot, freely mixed).
- Stage 2 documented with both decoders, n_sec.mode, teacher forcing,
stage1_context and the three particle_type.target options.
- Roadmap: v0.3.0 recorded as implemented/released; WGAN and MoE routing
as implemented but unvalidated, with the router retrain as next step.
- Analysis: run dir is <cwd>/analysis_runs/analysis_<id>, plus
variables/reduced/runtime_estimate and analyze list/merge-one/metrics.
- Added giant model summary, configs/, and the CI-automated version and
changelog bump.
README drift fixes only: project tree for the model/analysis/training
splits, analyze run-dir default, missing subcommands, --precision and
--stage2-stage1-context, the extras list, and two accuracy fixes
(--router configures stage 1 only; --conditioning sets two independent
axes at once).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The automated changelog (gitea #50) deliberately started fresh with no
backfill; this reverses that call now that it's wanted. v0.2.0-v0.3.2 are
generated from tag history via git-cliff/cliff.toml, matching the format of
existing entries. v0.3.3 was bumped but never tagged, so its commits stay
folded into the existing v0.3.4 entry. The v0.2.0 range (198 uncurated
pre-automation commits) is hand-curated to drop duplicate commits and
dev-log noise (WIP markers, incomplete-validation runs, repeated
"Apply ruff format").
shower_containment_depth_90/95's title contains a literal "%" (e.g.
"...(90% of deposited energy)"), which usetex reads as a comment marker
and aborts LaTeX compilation. Since render_all processes reduced JSON
files in sorted filename order, this killed every plot id sorting after
these two in the same run.
Escape title/xlabel once, centrally, in render()'s dispatch (the one
place every renderer kind draws them from before handing off to
plotstyle/matplotlib) rather than at each catalog.py call site, so any
future catalog title with a %, &, #, etc. is covered automatically.
_plot_metadata keeps using the unescaped Reduced for the gallery YAML,
since that's not LaTeX.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant analyze compares N rollout YAMLs against one shared reference file
(all must name the same dataset, checked up front) instead of exactly one
rollout vs one reference, rendering each rollout as its own colored series
against a single reference line/panel. Series names come from a repeated
--label flag, else the YAML stem, else "rollout" for a single YAML — a
single-rollout run keeps rendering identically to before this change.
Bundle now holds a name-keyed dict of rollout sides instead of one fixed
pair, every catalog compute_partial/finalize builds a Reduced.payload
keyed the same way ("series": {name: ...}, "reference": ... as the one
distinguished non-rollout entry), and every renderer draws N series (or
N panels, for the two heatmap-shaped specs and the router/type-embedding
diagnostics, which are inherently one-matrix/one-checkpoint per rollout)
against the reference's fixed dashed-ink style.
CliRunner stores an uncaught exception in result.exception, not
result.output, so the skip condition never matched and the test
failed outright on CI machines without LaTeX installed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Picks 4 of the 7 catalog additions the issue proposed (the smaller-lift
ones; 2D joint plots, PIT calibration, and the throughput/accuracy scatter
are left for follow-up issues):
- marginal_distance_summary: a var x grouping-axis KS-statistic heatmap,
reusing the existing marginal hist1d compute and just adding a finalize —
a single at-a-glance regression scorecard instead of N overlay plots.
- n_sec_confusion: predicted (rollout) vs true (reference) secondary count
per event, paired by event_id since a rollout is seeded from the same
events as its reference file. Needed a new zero-filling primitive
(reduce.sec_count_by_event) since a plain group_by over secondary rows
silently drops zero-secondary events.
- shower_containment_depth_{90,95}: per-event depth containing 90%/95% of
deposited energy, derived from the same per-event depth-bin matrix the
longitudinal profile already computes.
- router_specialization: max gate weight vs energy per side, summarizing
router_gating's full stacked area into the one trend line the roadmap's
MoE writeup describes (the ~60-65% ceiling), to make a future
lambda_balance>0 retrain's effect on specialization checkable at a glance.
Both new heatmap-shaped plots (distance summary, confusion matrix) share one
new "heatmap" Reduced kind/renderer rather than two near-identical ones.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
MetricsCollector writes one row per epoch to <run_dir>/metrics.csv, but
nothing read or plotted it. giant/training/plots.py reads the CSV header
dynamically (the column set varies by run: flow/ddpm vs wgan, routed vs
not) and renders loss/lr/accuracy/grad-norm/router/wgan-balance/throughput
plots with the same plotstyle conventions giant/analysis/render.py uses,
skipping any figure whose columns aren't present for a given run.
Wired up as `giant analyze metrics <run_dir>`, writing PDFs into the same
gitignored analysis_runs/ directory `analyze prep`/`submit` already use
(derive_metrics_dir mirrors derive_run_dir) rather than into the training
run directory itself.
CriticModel was the one stage-shaped class left out of the trunk-registry
(gitea #33), block-conditioning-registry (gitea #34), and StageModel-base
(gitea #39) refactors: it hand-rolled a plain ResBlock stack, so a
routed/FiLM/AdaLN trunk was available to every generative stage model except
the critic competing against them under WGAN-GP.
CriticModel now subclasses StageModel (reusing its cond_enc construction, and
a stage-2 context-fusion helper factored out of Stage2OneShot onto the base)
and builds its body via build_trunk (output width 1) instead of a bespoke
ResBlock loop, so trunk.type/trunk.block_conditioning now affect the critic
too. Each stage's critic inherits its own generator's trunk config rather
than a new critic_trunk config key, mirroring the existing
critic_hidden_dim/critic_n_res_blocks "0 = inherit from generator" pattern.
Router mixing (MoE) for the critic stays out of scope. Since CriticModel is
training-only and never persisted for inference, and WGAN-GP is still
unbenchmarked, its state_dict shape has no back-compat burden.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Turns "is this component earning its parameters?" into a one-line
config flip for each of the three pluggable network components:
- router.type = "none" (NoneRouter, giant/model/routers.py): still
builds n_experts expert trunks via RoutedTrunk, but replaces the
learned gate with a uniform 1/n_experts weight for every row — no
centers/embeddings/classifier. Distinct from router.enabled=false
(which drops routing/mixing entirely): this isolates whether the
*learned routing signal* specifically is earning its parameters,
holding expert count fixed.
- stage2_model.autoregressive.history = "none" (NoHistory,
giant/model/history.py): ignores feat/has_prev entirely and always
returns zeros, ablating whether the AR decoder's history
conditioning earns its parameters. Already validated for free by
gitea #35's generic HISTORY_REGISTRY membership check.
- trunk.type = "linear" (LinearTrunk, giant/model/trunks.py): a bare
nn.Linear(in_dim + cond_dim, out_dim) body, no ResBlock stack. Per
gitea #33's design, this composes for free with router.enabled=true
("mixture of trivial linear experts").
Both blocking issues (#33 trunk registry, #35 pluggable history
encoder) are closed, so this was unblocked.
The prior e2e run (task 1348) failed on the bump-version job's push step
with a 403 Forbidden — CI_TOKEN lacked write access. Note this on the
checkout step so the requirement isn't lost, now that the token has been
rescoped. Trivial commit to re-open a merge request and re-run the job
end to end.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Version bumps and release tags were entirely manual; the only CI automation
was sync-version-on-tag, which corrects pyproject.toml if a hand-pushed tag
drifted. This flips that: a new bump-version job (needs the four existing
checks, gated to actual merge commits on master via HEAD^@'s parent count so
direct/squash/rebase pushes are untouched) uses bump-my-version to auto-bump
the patch version when a merged branch didn't already bump it itself, then
generates a changelog entry with git-cliff and pushes a matching vX.Y.Z tag.
git-cliff's cliff.toml is tuned to this repo's plain imperative commit style
(no feat:/fix: prefixes): commits are grouped Added/Fixed/Removed/Changed by
leading verb, "(gitea #N)" is linkified, and merge/[skip ci] commits are
dropped. Per user decision during planning: the changelog generator folds in
@lars's comment on the issue (asking to fold in changelog generation rather
than deferring it), and CHANGELOG.md starts fresh with no backfill of
v0.2.0-v0.3.3.
sync-version-on-tag is left untouched as the safety net for hand-tagging.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The v0.3.0 pivot exists because the 2026-08-03 WGAN rollout benchmark
produced zero photon secondaries and ~4M hallucinated antineutrinos —
even with a correctly-sized top-N species vocabulary (gitea #29), plain
cross-entropy over a class distribution spanning orders of magnitude
still under-predicts rare-but-physical species.
stage2_model.particle_type.class_weighting = "none" | "inverse_freq"
(default "none", fully back-compat) weights the stage-2 type head's CE
loss (FlowDDPMStageTrainer._type_loss) by inverse class frequency,
normalized to mean 1 so switching it on doesn't rescale the type loss
against particle_type.lambda / the generator loss it's summed with.
The per-class counts the weighting needs don't already exist despite the
issue's premise: _topn_plus_other_map (giant/data/loader.py) previously
kept counts only for keys folded into "other", dropping the kept classes'
counts on the floor. TopNMap now carries class_counts (index -> count),
round-tripped through the setup-cache sidecar (format version bumped
3->4, since existing sidecars have none) and through checkpoints
(tolerantly — a pre-#44 checkpoint decodes to {}, since only training-time
loss weighting reads it, not inference).
Decisions made during planning (with the user): dropped "effective_num"
from the issue's proposed three-way enum (no beta hyperparameter to
design around) — final domain is "none" | "inverse_freq". Weights are
mean-1-normalized. validate_config rejects class_weighting != "none"
combined with particle_type.target != "onehot" or
stage2_model.generator == "wgan" (both have no class CE to weight),
following the #28/#30 dead-key-must-not-go-silent convention. Branch
fix/issue-44 off master.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant/analysis/sources.py's open_side scanned a reference directory of
parquet shards with a bare glob and never offset event_id across them.
Each shard is a separate Geant4 job whose own event_id numbering restarts
from 0, so events from different shards collided on the same event_id,
corrupting every downstream per-event grouping and the event_id % n_chunks
condor chunking — the same root cause already fixed on the training/rollout
side via giant/data/loader.py's per-file event_id_offset.
open_side's reference branch now uses find_parquet_files (the same
deterministically ordered file lister giant rollout's _seed_from_data uses)
and offsets each shard's event_id via a join on polars' include_file_paths,
so both sides of a comparison agree on what an event_id means. Two
incidental behaviour changes come along for free: .manifest references now
work (they crashed before), and the directory glob narrows from recursive
**/*.parquet to top-level *.parquet, matching the file list rollout itself
used to assign offsets — a deliberate choice, since a differing file list
would make the two sides' offsets disagree again in a subtler way.
No overflow guard on the per-shard offset stride (unlike loader's
_offset_event_id): checking it here would cost an eager event_id-column
read per shard in every condor compute job, and giant rollout already runs
that check over the same file list when producing the seed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars
gitea-actions