24 Commits

Author SHA1 Message Date
gitea-actions 1675052ecd chore: update changelog for v0.3.5 [skip ci] 2026-08-24 07:37:23 +00:00
gitea-actions eb9d331bea chore: bump version 0.3.4 -> 0.3.5 [skip ci] 2026-08-24 07:37:22 +00:00
lars 12689cf5b6 Merge pull request 'Add "none" variants for router, history, and trunk (gitea #45)' (#73) from fix/issue-45 into master
CI / Lint (ruff check) (push) Successful in 30s
CI / Format (ruff format) (push) Successful in 29s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 34s
CI / Tests (push) Successful in 2m47s
CI / Bump version, tag, and update changelog on merge to master (push) Successful in 42s
Reviewed-on: #73
2026-08-24 09:32:30 +02:00
lars 732d5f1cd2 Add "none" variants for router, history, and trunk (gitea #45)
CI / Format (ruff format) (push) Successful in 30s
CI / Lint (ruff check) (push) Successful in 34s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 42s
CI / Lint (ruff check) (pull_request) Successful in 41s
CI / Format (ruff format) (pull_request) Successful in 46s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 43s
CI / Tests (push) Successful in 5m12s
CI / Tests (pull_request) Successful in 5m10s
CI / Bump version, tag, and update changelog on merge to master (push) Has been skipped
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
Turns "is this component earning its parameters?" into a one-line
config flip for each of the three pluggable network components:

- router.type = "none" (NoneRouter, giant/model/routers.py): still
  builds n_experts expert trunks via RoutedTrunk, but replaces the
  learned gate with a uniform 1/n_experts weight for every row — no
  centers/embeddings/classifier. Distinct from router.enabled=false
  (which drops routing/mixing entirely): this isolates whether the
  *learned routing signal* specifically is earning its parameters,
  holding expert count fixed.

- stage2_model.autoregressive.history = "none" (NoHistory,
  giant/model/history.py): ignores feat/has_prev entirely and always
  returns zeros, ablating whether the AR decoder's history
  conditioning earns its parameters. Already validated for free by
  gitea #35's generic HISTORY_REGISTRY membership check.

- trunk.type = "linear" (LinearTrunk, giant/model/trunks.py): a bare
  nn.Linear(in_dim + cond_dim, out_dim) body, no ResBlock stack. Per
  gitea #33's design, this composes for free with router.enabled=true
  ("mixture of trivial linear experts").

Both blocking issues (#33 trunk registry, #35 pluggable history
encoder) are closed, so this was unblocked.
2026-08-24 09:22:36 +02:00
gitea-actions ef8a2f4e55 chore: update changelog for v0.3.4 [skip ci] 2026-08-23 19:50:10 +00:00
gitea-actions d61a9b7661 chore: bump version 0.3.3 -> 0.3.4 [skip ci] 2026-08-23 19:50:08 +00:00
lars dc16265e18 Merge pull request 'Document CI_TOKEN's write:repository scope requirement (gitea #50)' (#72) from fix/issue-50 into master
CI / Lint (ruff check) (push) Successful in 34s
CI / Format (ruff format) (push) Successful in 32s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 37s
CI / Tests (push) Successful in 2m43s
CI / Bump version, tag, and update changelog on merge to master (push) Successful in 41s
Reviewed-on: #72
2026-08-23 21:39:52 +02:00
lars aff0ef881f Document CI_TOKEN's write:repository scope requirement (gitea #50)
CI / Lint (ruff check) (push) Successful in 31s
CI / Format (ruff format) (push) Successful in 31s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 44s
CI / Type check (ty) (push) Successful in 48s
CI / Format (ruff format) (pull_request) Successful in 33s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 35s
CI / Tests (push) Successful in 5m12s
CI / Tests (pull_request) Successful in 4m40s
CI / Bump version, tag, and update changelog on merge to master (push) Has been skipped
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
The prior e2e run (task 1348) failed on the bump-version job's push step
with a 403 Forbidden — CI_TOKEN lacked write access. Note this on the
checkout step so the requirement isn't lost, now that the token has been
rescoped. Trivial commit to re-open a merge request and re-run the job
end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-23 21:38:41 +02:00
lars d0cbcbce80 Merge pull request 'Auto-bump patch version, tag, and update changelog on merge to master (gitea #50)' (#71) from fix/issue-50 into master
CI / Type check (ty) (push) Successful in 36s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (push) Successful in 28s
CI / Format (ruff format) (push) Successful in 36s
CI / Tests (push) Successful in 3m0s
CI / Bump version, tag, and update changelog on merge to master (push) Failing after 44s
Reviewed-on: #71
2026-08-18 10:50:16 +02:00
lars 10a57322f9 Merge branch 'master' into fix/issue-50
CI / Format (ruff format) (push) Successful in 36s
CI / Lint (ruff check) (push) Successful in 37s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 43s
CI / Type check (ty) (push) Successful in 45s
CI / Format (ruff format) (pull_request) Successful in 39s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 39s
CI / Tests (pull_request) Successful in 4m54s
CI / Tests (push) Successful in 5m2s
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
CI / Bump version, tag, and update changelog on merge to master (push) Has been skipped
2026-08-18 10:42:32 +02:00
lars c09ebd2410 Merge pull request 'Add class-balanced secondary particle-type loss (gitea #44)' (#70) from fix/issue-44 into master
CI / Format (ruff format) (push) Successful in 34s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 34s
CI / Lint (ruff check) (push) Successful in 25s
CI / Tests (push) Successful in 4m29s
Reviewed-on: #70
2026-08-18 10:41:42 +02:00
lars 5b478d2831 Auto-bump patch version, tag, and update changelog on merge to master (gitea #50)
Version bumps and release tags were entirely manual; the only CI automation
was sync-version-on-tag, which corrects pyproject.toml if a hand-pushed tag
drifted. This flips that: a new bump-version job (needs the four existing
checks, gated to actual merge commits on master via HEAD^@'s parent count so
direct/squash/rebase pushes are untouched) uses bump-my-version to auto-bump
the patch version when a merged branch didn't already bump it itself, then
generates a changelog entry with git-cliff and pushes a matching vX.Y.Z tag.

git-cliff's cliff.toml is tuned to this repo's plain imperative commit style
(no feat:/fix: prefixes): commits are grouped Added/Fixed/Removed/Changed by
leading verb, "(gitea #N)" is linkified, and merge/[skip ci] commits are
dropped. Per user decision during planning: the changelog generator folds in
@lars's comment on the issue (asking to fold in changelog generation rather
than deferring it), and CHANGELOG.md starts fresh with no backfill of
v0.2.0-v0.3.3.

sync-version-on-tag is left untouched as the safety net for hand-tagging.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-18 10:40:19 +02:00
lars de805fb0a7 Merge pull request 'Offset event_id across multi-shard reference reads in giant analyze (gitea #22)' (#69) from fix/issue-22 into master
CI / Lint (ruff check) (push) Successful in 32s
CI / Format (ruff format) (push) Successful in 30s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 34s
CI / Tests (push) Failing after 12m16s
Reviewed-on: #69
2026-08-18 10:23:20 +02:00
lars fce47b128c Add class-balanced secondary particle-type loss (gitea #44)
CI / Lint (ruff check) (push) Successful in 36s
CI / Format (ruff format) (push) Successful in 38s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 38s
CI / Format (ruff format) (pull_request) Successful in 43s
CI / Lint (ruff check) (pull_request) Successful in 45s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 47s
CI / Tests (push) Successful in 5m20s
CI / Tests (pull_request) Successful in 4m50s
The v0.3.0 pivot exists because the 2026-08-03 WGAN rollout benchmark
produced zero photon secondaries and ~4M hallucinated antineutrinos —
even with a correctly-sized top-N species vocabulary (gitea #29), plain
cross-entropy over a class distribution spanning orders of magnitude
still under-predicts rare-but-physical species.

stage2_model.particle_type.class_weighting = "none" | "inverse_freq"
(default "none", fully back-compat) weights the stage-2 type head's CE
loss (FlowDDPMStageTrainer._type_loss) by inverse class frequency,
normalized to mean 1 so switching it on doesn't rescale the type loss
against particle_type.lambda / the generator loss it's summed with.

The per-class counts the weighting needs don't already exist despite the
issue's premise: _topn_plus_other_map (giant/data/loader.py) previously
kept counts only for keys folded into "other", dropping the kept classes'
counts on the floor. TopNMap now carries class_counts (index -> count),
round-tripped through the setup-cache sidecar (format version bumped
3->4, since existing sidecars have none) and through checkpoints
(tolerantly — a pre-#44 checkpoint decodes to {}, since only training-time
loss weighting reads it, not inference).

Decisions made during planning (with the user): dropped "effective_num"
from the issue's proposed three-way enum (no beta hyperparameter to
design around) — final domain is "none" | "inverse_freq". Weights are
mean-1-normalized. validate_config rejects class_weighting != "none"
combined with particle_type.target != "onehot" or
stage2_model.generator == "wgan" (both have no class CE to weight),
following the #28/#30 dead-key-must-not-go-silent convention. Branch
fix/issue-44 off master.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 23:02:50 +02:00
lars c1e6ffd8c6 Offset event_id across multi-shard reference reads in giant analyze (gitea #22)
CI / Lint (ruff check) (push) Successful in 34s
CI / Format (ruff format) (push) Successful in 34s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 28s
CI / Type check (ty) (push) Successful in 30s
CI / Format (ruff format) (pull_request) Successful in 38s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 39s
CI / Tests (push) Successful in 4m54s
CI / Tests (pull_request) Successful in 4m55s
giant/analysis/sources.py's open_side scanned a reference directory of
parquet shards with a bare glob and never offset event_id across them.
Each shard is a separate Geant4 job whose own event_id numbering restarts
from 0, so events from different shards collided on the same event_id,
corrupting every downstream per-event grouping and the event_id % n_chunks
condor chunking — the same root cause already fixed on the training/rollout
side via giant/data/loader.py's per-file event_id_offset.

open_side's reference branch now uses find_parquet_files (the same
deterministically ordered file lister giant rollout's _seed_from_data uses)
and offsets each shard's event_id via a join on polars' include_file_paths,
so both sides of a comparison agree on what an event_id means. Two
incidental behaviour changes come along for free: .manifest references now
work (they crashed before), and the directory glob narrows from recursive
**/*.parquet to top-level *.parquet, matching the file list rollout itself
used to assign offsets — a deliberate choice, since a differing file list
would make the two sides' offsets disagree again in a subtler way.

No overflow guard on the per-shard offset stride (unlike loader's
_offset_event_id): checking it here would cost an eager event_id-column
read per shard in every condor compute job, and giant rollout already runs
that check over the same file list when producing the seed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 15:53:02 +02:00
lars f60af64d00 Merge pull request 'Add bf16 autocast to the training loop (gitea #47)' (#68) from fix/issue-47 into master
CI / Lint (ruff check) (push) Successful in 28s
CI / Format (ruff format) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 24s
CI / Tests (push) Successful in 2m37s
Reviewed-on: #68
2026-08-17 15:45:51 +02:00
lars 78978769f6 Add bf16 autocast to the training loop (gitea #47)
CI / Lint (ruff check) (push) Successful in 28s
CI / Format (ruff format) (push) Successful in 30s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 44s
CI / Lint (ruff check) (pull_request) Successful in 41s
CI / Format (ruff format) (pull_request) Successful in 31s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 32s
CI / Tests (pull_request) Successful in 4m36s
CI / Tests (push) Successful in 4m49s
giant/ had no autocast/GradScaler/torch.compile anywhere despite the
project's ~10x-native-Geant4 eval-budget target. This adds bf16 mixed
precision to the training step (both FlowDDPMStageTrainer and
WGANStageTrainer) via a new train.precision config key ("fp32" default,
"bf16" opt-in) and giant.training.amp.resolve_autocast.

torch.compile is a separate, much larger surface (data-dependent routed
dispatch, the autoregressive sampler's per-token control flow, arbitrary
rollout batch sizes) and is left for a follow-up issue, per discussion.

Scope decisions made during planning:
- fp32 + bf16 only, no fp16/GradScaler. fp16 breaks two things in this
  codebase: routers.py's three 1e-8 epsilons sit below fp16's ~6e-8
  subnormal floor, and gradient_penalty's grad norm overflows fp16's
  range at ordinary early-WGAN-GP gradient magnitudes. Every training
  GPU in the fleet (A100/L40S/H200/RTX 4070) has native bf16; only
  pre-Ampere V100s would need fp16.
- resolve_autocast raises loudly if bf16 is requested on hardware that
  can't do it, rather than silently falling back to fp32.
- Autocast wraps the training step only; val_loss (and the
  best-checkpoint selection it drives) stays fp32 so it's comparable
  across every run recorded so far.
- _route_forward's mixture accumulator (giant/model/trunks.py) was a
  hard-fp32 torch.zeros with no dtype, so under autocast a RoutedTrunk
  silently returned a different output dtype than an unrouted
  ExpertTrunk purely because router.enabled was set. Fixed to match the
  experts' own dtype; the gate weights (forced fp32 for their own
  numerical stability) are cast down before combining, so the
  mixture's numerics stay solid without reintroducing the dtype split.
- Added explicit fp32 guards (autocast(enabled=False)) around spots
  that are correct in fp32 but degrade quietly rather than crash in
  bf16: the router's balance/entropy losses and gate softmax, the
  stage-2 stick-breaking cumprod, and gradient_penalty's
  double-backward + grad norm.

Benchmarked on the local RTX 4070 against configs/baseline.toml's
hyperparams (hidden_dim 512/6 blocks, bs 4096) on a synthetic dataset:
bf16 gave 1.05-1.35x training throughput and 18-33% lower peak GPU
memory across one-shot/routed/autoregressive stage-2 configs, with the
autoregressive path (the dominant cost per baseline.toml) benefiting
most on both axes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 15:36:12 +02:00
lars 692acd77eb Merge pull request 'Add per-stage init_from/freeze (gitea #42)' (#67) from fix/issue-42 into master
CI / Format (ruff format) (push) Successful in 28s
CI / Lint (ruff check) (push) Successful in 28s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 30s
CI / Tests (push) Successful in 2m35s
Reviewed-on: #67
2026-08-17 14:48:38 +02:00
lars e8842c56d7 Bump patch version to 0.3.3
CI / Lint (ruff check) (push) Successful in 28s
CI / Format (ruff format) (push) Successful in 29s
CI / Sync project version with tag (push) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 31s
CI / Type check (ty) (push) Successful in 38s
CI / Format (ruff format) (pull_request) Successful in 41s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 40s
CI / Tests (pull_request) Successful in 4m59s
CI / Tests (push) Successful in 5m11s
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 14:25:51 +02:00
lars 87e37ebe14 Add per-stage init_from/freeze (gitea #42)
CI / Format (ruff format) (push) Successful in 29s
CI / Lint (ruff check) (push) Successful in 30s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 38s
CI / Lint (ruff check) (pull_request) Successful in 31s
CI / Format (ruff format) (pull_request) Successful in 44s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 47s
CI / Tests (push) Successful in 4m32s
CI / Tests (pull_request) Successful in 4m25s
stage{1,2}_model.active = false already trains one stage alone, but the
checkpoint it writes holds only that stage, so giant rollout refuses it --
the "retrain stage 2 alone against a fixed, known-good stage 1" experiment
the 2026-08-03 species failure calls for wasn't runnable end to end.

Adds stage{1,2}_model.init_from (a checkpoint .pt to load this stage's
weights from before training) and .freeze (never update them), symmetric
across both stages. Both stages stay active = true, so both get built and
both land in the output checkpoint -- the frozen stage is merely
initialized from disk instead of from scratch.

Decisions made during planning:
- Soft freeze: forward/backward still run every batch (loss/grad_norm stay
  meaningful, no autograd special-casing), only optimizer.step() (and, for
  the frozen stage, lr_sched.step()/EMA update) is skipped -- weights are
  byte-identical for the whole run. This is StageTrainer._step_optimizer,
  shared by the non-adversarial path and both halves (generator + critic)
  of the WGAN path, so a frozen WGAN stage's critic freezes too.
- validate_config requires init_from whenever freeze = true, unless the run
  is a --resume (a resumed frozen stage's weights come from the resume
  checkpoint instead) -- freezing a randomly-initialized model is almost
  certainly a mistake.
- CLI flags on both `giant train` and `giant new-run`
  (--stage{1,2}-init-from/--stage{1,2}-freeze), matching every other
  per-stage model knob's existing treatment.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 14:23:20 +02:00
lars 8290e350b8 Merge pull request 'Implement stage2_model.stage1_context = "sampled" (gitea #41)' (#66) from fix/issue-41 into master
CI / Lint (ruff check) (push) Successful in 30s
CI / Format (ruff format) (push) Successful in 29s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 25s
CI / Tests (push) Successful in 2m37s
Reviewed-on: #66
2026-08-17 13:40:54 +02:00
lars 48faaee79d Implement stage2_model.stage1_context = "sampled" (gitea #41)
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 30s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 31s
CI / Tests (push) Successful in 2m26s
CI / Lint (ruff check) (pull_request) Successful in 27s
CI / Format (ruff format) (pull_request) Successful in 27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 27s
CI / Tests (pull_request) Successful in 2m23s
Stage 2 was trained on ground-truth stage-1 outcomes but deployed on
sampled ones, and in a rollout that gap compounds over every step of
every track — the same train/inference gap teacher_forcing="scheduled"
already closes within stage 2, just never applied at the stage
boundary. "sampled" was declared in the schema but rejected loudly by
validate_config as unimplemented; this lands the real implementation.

Mirrors the existing scheduled-sampling precedent rather than a hard
switch: new stage2_model.ctx_p_start/ctx_p_end (defaults 1.0 -> 0.0)
linearly ramp P(condition on ground truth) from epoch 0 to the final
epoch, so stage 2 doesn't chase a wildly moving stage-1 target early in
training. Per the plan discussed with the user: the sample is drawn
from stage 1's sampling_model() (EMA weights when present, matching
what inference actually deploys), mixed per example via a Bernoulli
draw (never blended within a row), and validation always uses the
ground truth regardless of the schedule. Fixes a latent bug the same
pattern would otherwise have hit: every sampler in giant/sample.py
flips its model to .eval() with no restore, so sampling from the raw
(non-EMA) stage-1 model mid-step now explicitly restores its .training
flag afterward to avoid silently corrupting stage 1's own training mode
for the rest of the epoch.

validate_config now enforces stage1_context in {"truth", "sampled"},
requires both stages active for "sampled" (nothing to sample from
otherwise), range-checks ctx_p_start/ctx_p_end, and rejects the
ctx_p_start = ctx_p_end = 1.0 configuration as an unadvertised no-op
identical to "truth".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 12:27:13 +02:00
lars 09bea2cbff Merge pull request 'Add giant model summary command (gitea #46)' (#65) from fix/issue-46 into master
CI / Lint (ruff check) (push) Successful in 27s
CI / Format (ruff format) (push) Successful in 29s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 24s
CI / Tests (push) Successful in 2m30s
Reviewed-on: #65
2026-08-17 12:08:59 +02:00
lars cc9646f279 Add giant model summary command (gitea #46)
CI / Lint (ruff check) (push) Successful in 37s
CI / Format (ruff format) (push) Successful in 38s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 27s
CI / Tests (push) Successful in 2m34s
CI / Lint (ruff check) (pull_request) Successful in 30s
CI / Format (ruff format) (pull_request) Successful in 30s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 29s
CI / Tests (pull_request) Successful in 2m35s
giant model summary --config config.toml builds the resolved Stage1/Stage2
graph from a config with no dataset attached (pdg_vocab/mat_vocab are
supplied as placeholders via --pdg-vocab/--mat-vocab, since the real
training vocab is dataset-derived) and prints per-module parameter counts,
trunk in/out widths, which heads exist, and which
conditioning/stage1_model/stage2_model config keys actually shaped the
build.

The consumed-keys half uses differential probing rather than static
identifier matching: build once for a fingerprint (submodule presence,
every parameter's/buffer's shape+dtype, every plain scalar attribute a
module stores on itself), then perturb one leaf at a time, rebuild, and
compare. A changed fingerprint (or a raise) means the key is consumed; no
change means it's inert *under this particular config* -- e.g. any
stage1_model.router.* key when router.enabled=false. A curated
_NOT_BUILD_TIME table separates keys legitimately owned by the
trainer/sampler/rollout (loss weights, WGAN-GP hyperparameters,
teacher-forcing schedules) from genuinely-inert ones, verified against
those call sites. A few config keys branch on equality against one specific
string literal (n_sec.owner=="stage1", n_sec.mode=="stop_token",
particle_type.target=="physical"); a single generic sentinel probe missed
all three since the config's current value and the sentinel landed in the
same branch, so those three leaves get their real alternative value tried
too (_STRING_ALTERNATIVES).

giant.config.leaf_paths is promoted out of
tests/test_config_consumed_keys.py (previously a private test-local
duplicate) so both audits -- the static per-identifier one and this new
runtime per-config one -- walk the exact same DEFAULT_CONFIG tree.
ExpertTrunk/RoutedTrunk now also expose in_dim (out_dim already existed),
needed to report trunk widths generically.

Decisions made during planning: --pdg-vocab/--mat-vocab default to 300 and
len(MATERIAL_PROPERTIES); the consumed-keys report is scoped to
conditioning/stage1_model/stage2_model only (train/meta are out of scope
for a model-only build); the module tree prints every submodule at any
depth.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-17 11:48:02 +02:00
39 changed files with 2838 additions and 179 deletions
+17
View File
@@ -0,0 +1,17 @@
[tool.bumpversion]
current_version = "0.3.5"
parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
serialize = ["{major}.{minor}.{patch}"]
search = "{current_version}"
replace = "{new_version}"
regex = false
allow_dirty = false
commit = true
tag = false
message = "chore: bump version {current_version} -> {new_version} [skip ci]"
pre_commit_hooks = ["uv lock", "git add uv.lock"]
[[tool.bumpversion.files]]
filename = "pyproject.toml"
search = "version = \"{current_version}\""
replace = "version = \"{new_version}\""
+86
View File
@@ -88,6 +88,92 @@ jobs:
name: coverage-report name: coverage-report
path: coverage.xml path: coverage.xml
bump-version:
name: Bump version, tag, and update changelog on merge to master
needs: [ruff-check, ruff-format, type-check, test]
if: github.ref == 'refs/heads/master' && github.event_name == 'push'
runs-on: ubuntu-latest
container:
image: docker.gitea.com/runner-images:ubuntu-latest
volumes:
- /srv/act-runner-cache/uv:/uv-cache
steps:
# CI_TOKEN needs write:repository scope (not just read) — this job
# pushes commits and tags to master, unlike ruff-check/ruff-format/
# type-check/test above, which only need to check out the repo.
- uses: actions/checkout@v4
with:
token: ${{ secrets.CI_TOKEN }}
fetch-depth: 0
- name: Check whether this push is a merge commit
id: merge_check
run: |
PARENTS=$(git rev-parse HEAD^@ | wc -l)
echo "HEAD has $PARENTS parent(s)"
if [ "$PARENTS" -ge 2 ]; then
echo "is_merge=true" >> "$GITHUB_OUTPUT"
else
echo "is_merge=false" >> "$GITHUB_OUTPUT"
fi
- uses: astral-sh/setup-uv@v5
if: steps.merge_check.outputs.is_merge == 'true'
with:
enable-cache: false
- run: |
echo "UV_CACHE_DIR=/uv-cache" >> "$GITHUB_ENV"
echo "UV_LINK_MODE=copy" >> "$GITHUB_ENV"
if: steps.merge_check.outputs.is_merge == 'true'
- run: uv sync --extra cpu --extra dev
if: steps.merge_check.outputs.is_merge == 'true'
- name: Configure git identity
if: steps.merge_check.outputs.is_merge == 'true'
run: |
git config user.name "gitea-actions"
git config user.email "actions@git.larsbogner.de"
- name: Bump patch version if this merge didn't already bump it
if: steps.merge_check.outputs.is_merge == 'true'
run: |
OLD_VERSION=$(git show "${{ github.event.before }}:pyproject.toml" 2>/dev/null | grep -m1 '^version = ' | sed -E 's/version = "(.*)"/\1/')
CURRENT_VERSION=$(uv version --short)
if [ -z "$OLD_VERSION" ]; then
echo "Could not read pyproject.toml at github.event.before; falling back to HEAD^1"
OLD_VERSION=$(git show "HEAD^1:pyproject.toml" | grep -m1 '^version = ' | sed -E 's/version = "(.*)"/\1/')
fi
if [ "$OLD_VERSION" = "$CURRENT_VERSION" ]; then
echo "Version unchanged by this merge ($CURRENT_VERSION); bumping patch"
uv run bump-my-version bump patch --current-version "$CURRENT_VERSION"
else
echo "Branch already bumped the version ($OLD_VERSION -> $CURRENT_VERSION); skipping auto-bump"
fi
- name: Update changelog for the current version if not already tagged
if: steps.merge_check.outputs.is_merge == 'true'
run: |
VERSION=$(uv version --short)
TAG="v$VERSION"
if git rev-parse "$TAG" >/dev/null 2>&1; then
echo "Tag $TAG already exists; skipping changelog update"
else
uv run git-cliff --tag "$TAG" --unreleased --prepend CHANGELOG.md
git add CHANGELOG.md
if ! git diff --cached --quiet -- CHANGELOG.md; then
git commit -m "chore: update changelog for $TAG [skip ci]"
else
git restore --staged CHANGELOG.md
fi
fi
- name: Push commits and tag the current version
if: steps.merge_check.outputs.is_merge == 'true'
run: |
git push origin HEAD:master
VERSION=$(uv version --short)
TAG="v$VERSION"
if git rev-parse "$TAG" >/dev/null 2>&1; then
echo "Tag $TAG already exists"
else
git tag -a "$TAG" -m "$TAG"
git push origin "refs/tags/$TAG"
fi
sync-version-on-tag: sync-version-on-tag:
name: Sync project version with tag name: Sync project version with tag
if: startsWith(github.ref, 'refs/tags/') if: startsWith(github.ref, 'refs/tags/')
+34
View File
@@ -0,0 +1,34 @@
# Changelog
## [0.3.5] - 2026-08-24
### Added
- Add "none" variants for router, history, and trunk [gitea #45](https://git.larsbogner.de/lars/giant/issues/45)
## [0.3.4] - 2026-08-23
### Added
- Add giant model summary command [gitea #46](https://git.larsbogner.de/lars/giant/issues/46)
- Add per-stage init_from/freeze [gitea #42](https://git.larsbogner.de/lars/giant/issues/42)
- Add bf16 autocast to the training loop [gitea #47](https://git.larsbogner.de/lars/giant/issues/47)
- Add class-balanced secondary particle-type loss [gitea #44](https://git.larsbogner.de/lars/giant/issues/44)
### Changed
- Implement stage2_model.stage1_context = "sampled" [gitea #41](https://git.larsbogner.de/lars/giant/issues/41)
- Bump patch version to 0.3.3
- Offset event_id across multi-shard reference reads in giant analyze [gitea #22](https://git.larsbogner.de/lars/giant/issues/22)
- Auto-bump patch version, tag, and update changelog on merge to master [gitea #50](https://git.larsbogner.de/lars/giant/issues/50)
- Document CI_TOKEN's write:repository scope requirement [gitea #50](https://git.larsbogner.de/lars/giant/issues/50)
# Changelog
+1
View File
@@ -139,6 +139,7 @@ Useful flags on `giant train`:
- `--router` / `--router-type` / `--n-experts` / `--router-axis` — MoE routing - `--router` / `--router-type` / `--n-experts` / `--router-axis` — MoE routing
- `--wandb` — log per-epoch metrics to Weights & Biases (needs `uv sync --extra wandb`); metric names are `<stage>/<split>/<metric>` plus an unprefixed run-level tail, all derived from `giant/training/trainers.py` `MetricSpec`s - `--wandb` — log per-epoch metrics to Weights & Biases (needs `uv sync --extra wandb`); metric names are `<stage>/<split>/<metric>` plus an unprefixed run-level tail, all derived from `giant/training/trainers.py` `MetricSpec`s
- `--no-cache-setup` / `--rebuild-setup-cache` — control the setup-stage sidecar cache (vocab maps, event split, normalizer stats); `dwarf warm-cache` precomputes it - `--no-cache-setup` / `--rebuild-setup-cache` — control the setup-stage sidecar cache (vocab maps, event split, normalizer stats); `dwarf warm-cache` precomputes it
- `--stage1-init-from`/`--stage2-init-from` (checkpoint `.pt`) + `--stage1-freeze`/`--stage2-freeze` — load a stage's weights from another checkpoint and never update them, so the other stage can be retrained alone against a fixed, known-good one while still producing a complete, rollout-capable checkpoint
Config-file-only knobs (no CLI flag — use `--config config.toml`): `stage2_model.autoregressive.teacher_forcing`/`.history`, `stage2_model.particle_type.target`. v0.2 flat-schema configs and checkpoints load fine (auto-migrated). Config-file-only knobs (no CLI flag — use `--config config.toml`): `stage2_model.autoregressive.teacher_forcing`/`.history`, `stage2_model.particle_type.target`. v0.2 flat-schema configs and checkpoints load fine (auto-migrated).
+52
View File
@@ -0,0 +1,52 @@
# git-cliff configuration — see https://git-cliff.org/docs/configuration
#
# Commit messages in this repo aren't Conventional Commits; they're plain
# imperative summaries like "Add class-balanced secondary particle-type loss
# (gitea #44)". Parsing here is tuned to that convention rather than to
# feat:/fix:-style prefixes.
[changelog]
header = "# Changelog\n\n"
body = """
{% if version %}\
## [{{ version | trim_start_matches(pat="v") }}] - {{ timestamp | date(format="%Y-%m-%d") }}
{% else %}\
## [Unreleased]
{% endif %}\
{% for group, commits in commits | group_by(attribute="group") %}
### {{ group | striptags | trim | upper_first }}
{% for commit in commits %}
- {{ commit.message | upper_first }}
{% endfor %}
{% endfor %}
"""
trim = true
render_always = true
postprocessors = []
[git]
conventional_commits = false
filter_unconventional = false
require_conventional = false
split_commits = false
# Keep only the commit subject (first line), then linkify "(gitea #N)".
commit_preprocessors = [
{ pattern = "(?s)\n.*", replace = "" },
{ pattern = "\\(gitea #(\\d+)\\)", replace = "[gitea #${1}](https://git.larsbogner.de/lars/giant/issues/${1})" },
]
protect_breaking_commits = false
commit_parsers = [
{ message = "^Merge ", skip = true },
{ message = "\\[skip ci\\]", skip = true },
{ message = "^Add", group = "<!-- 0 -->Added" },
{ message = "^(Fix|Clamp|Clip)", group = "<!-- 1 -->Fixed" },
{ message = "^(Remove|Drop|Deprecate)", group = "<!-- 2 -->Removed" },
{ message = ".*", group = "<!-- 3 -->Changed" },
]
filter_commits = false
link_parsers = []
use_branch_tags = false
topo_order = false
topo_order_commits = true
sort_commits = "oldest"
recurse_submodules = false
+33 -3
View File
@@ -40,6 +40,11 @@ from giant.constants import (
TERM_MAX_STEPS, TERM_MAX_STEPS,
TERM_UNKNOWN_PDG, TERM_UNKNOWN_PDG,
) )
from giant.data.loader import event_id_offset, find_parquet_files
# Helper column name for the per-shard offset join in open_side; dropped before
# the LazyFrame is returned, so it never leaks into a caller's schema.
_SOURCE_PATH_COL = "__source_path"
# The world-frame physical columns both sides share under identical names. # The world-frame physical columns both sides share under identical names.
PHYS_COLS: tuple[str, ...] = ( PHYS_COLS: tuple[str, ...] = (
@@ -108,6 +113,22 @@ def open_side(source: str | Path | pl.LazyFrame, side: Side) -> pl.LazyFrame:
reference file's upstream ROOT→parquet conversion don't agree on integer reference file's upstream ROOT→parquet conversion don't agree on integer
width, and an uncast mismatch only surfaces later as a ``pl.concat`` width, and an uncast mismatch only surfaces later as a ``pl.concat``
``SchemaError`` (e.g. in ``build_context``'s pdg-count merge). ``SchemaError`` (e.g. in ``build_context``'s pdg-count merge).
The reference (a rollout's seed ``dataset``) may be a directory of parquet
shards, or a ``.manifest`` naming a subset, rather than a single file — each
such shard is a separate Geant4 job whose own ``event_id`` numbering
restarts from 0, so a multi-shard load offsets every shard's ids by
``giant.data.loader.event_id_offset(file_index)`` to keep them globally
unique, exactly as the training/rollout data pipeline already does
(``giant/data/loader.py``). ``file_index`` comes from
``find_parquet_files``'s deterministic ordering — the same list and
ordering ``giant rollout`` used (via ``_seed_from_data``) to offset the
rollout side's own ``event_id``s, so both sides agree on what an
``event_id`` means. There is no overflow guard here (unlike
``loader._offset_event_id``): checking it would cost an eager
``event_id``-column read per shard in every condor compute job, and
``giant rollout`` already ran that check over this exact file list when it
produced the seed.
""" """
if isinstance(source, pl.LazyFrame): if isinstance(source, pl.LazyFrame):
return source.with_columns(pl.col("pdg").cast(pl.Int64)) return source.with_columns(pl.col("pdg").cast(pl.Int64))
@@ -116,9 +137,18 @@ def open_side(source: str | Path | pl.LazyFrame, side: Side) -> pl.LazyFrame:
_check_rollout_metadata(path) _check_rollout_metadata(path)
lf = pl.scan_parquet(path) lf = pl.scan_parquet(path)
else: else:
# The reference (a rollout's seed `dataset`) may be a directory of files = find_parquet_files(path)
# parquet shards rather than a single file — scan them all. if len(files) == 1:
lf = pl.scan_parquet(str(path / "**/*.parquet")) if path.is_dir() else pl.scan_parquet(path) lf = pl.scan_parquet(files[0])
else:
offsets = {str(p): event_id_offset(i) for i, p in enumerate(files)}
lf = (
pl.scan_parquet(files, include_file_paths=_SOURCE_PATH_COL)
.with_columns(
pl.col("event_id") + pl.col(_SOURCE_PATH_COL).replace_strict(offsets, return_dtype=pl.Int64)
)
.drop(_SOURCE_PATH_COL)
)
return lf.with_columns(pl.col("pdg").cast(pl.Int64)) return lf.with_columns(pl.col("pdg").cast(pl.Int64))
+100 -1
View File
@@ -41,6 +41,7 @@ from giant.data.transforms import (
) )
from giant.checkpoint_io import CheckpointCompatibilityError, load_for_inference from giant.checkpoint_io import CheckpointCompatibilityError, load_for_inference
from giant.geometry import GeometryOracle from giant.geometry import GeometryOracle
from giant.materials import MATERIAL_PROPERTIES
from giant.pipeline import run_train_job from giant.pipeline import run_train_job
from giant.rollout import ( from giant.rollout import (
L1DistCollector, L1DistCollector,
@@ -487,6 +488,36 @@ def train(
help="WGAN-GP (--mode wgan only): critic depth for stage 2 (default: same as generator's n_res_blocks)", help="WGAN-GP (--mode wgan only): critic depth for stage 2 (default: same as generator's n_res_blocks)",
), ),
] = None, ] = None,
stage1_init_from: Annotated[
Optional[Path],
typer.Option(
"--stage1-init-from",
help="Checkpoint .pt to load stage 1's weights from before training starts "
"(gitea #42) — combine with --stage1-freeze to retrain stage 2 alone "
"against a fixed, known-good stage 1",
),
] = None,
stage1_freeze: Annotated[
Optional[bool],
typer.Option(
"--stage1-freeze/--no-stage1-freeze",
help="Never update stage 1's weights (requires --stage1-init-from, or --resume)",
),
] = None,
stage2_init_from: Annotated[
Optional[Path],
typer.Option(
"--stage2-init-from",
help="Checkpoint .pt to load stage 2's weights from before training starts (gitea #42)",
),
] = None,
stage2_freeze: Annotated[
Optional[bool],
typer.Option(
"--stage2-freeze/--no-stage2-freeze",
help="Never update stage 2's weights (requires --stage2-init-from, or --resume)",
),
] = None,
val_fraction: Annotated[Optional[float], typer.Option("--val-fraction", "-f")] = None, val_fraction: Annotated[Optional[float], typer.Option("--val-fraction", "-f")] = None,
seed: Annotated[ seed: Annotated[
Optional[int], Optional[int],
@@ -581,6 +612,14 @@ def train(
"steps (default: 50); per-epoch metrics always log in full", "steps (default: 50); per-epoch metrics always log in full",
), ),
] = None, ] = None,
precision: Annotated[
Optional[str],
typer.Option(
"--precision",
help="Training-step autocast precision: 'fp32' (default) or "
"'bf16'. No 'fp16' — see giant.training.amp.resolve_autocast",
),
] = None,
) -> None: ) -> None:
"""Train the GIANT surrogate model.""" """Train the GIANT surrogate model."""
batch_size_auto = False batch_size_auto = False
@@ -616,6 +655,7 @@ def train(
"wandb_project": wandb_project, "wandb_project": wandb_project,
"wandb_run_name": wandb_run_name, "wandb_run_name": wandb_run_name,
"wandb_log_every": wandb_log_every, "wandb_log_every": wandb_log_every,
"precision": precision,
"hidden_dim": hidden_dim, "hidden_dim": hidden_dim,
"n_blocks": n_blocks, "n_blocks": n_blocks,
"dropout": dropout, "dropout": dropout,
@@ -651,11 +691,15 @@ def train(
"stage1_critic_n_res_blocks": stage1_critic_n_res_blocks, "stage1_critic_n_res_blocks": stage1_critic_n_res_blocks,
"stage2_critic_hidden_dim": stage2_critic_hidden_dim, "stage2_critic_hidden_dim": stage2_critic_hidden_dim,
"stage2_critic_n_res_blocks": stage2_critic_n_res_blocks, "stage2_critic_n_res_blocks": stage2_critic_n_res_blocks,
"stage1_init_from": str(stage1_init_from) if stage1_init_from is not None else None,
"stage1_freeze": stage1_freeze,
"stage2_init_from": str(stage2_init_from) if stage2_init_from is not None else None,
"stage2_freeze": stage2_freeze,
} }
overrides = gconfig.overrides_from_flags(flag_values) overrides = gconfig.overrides_from_flags(flag_values)
cfg = gconfig.merge_cli_overrides(gconfig.DEFAULT_CONFIG, config, overrides) cfg = gconfig.merge_cli_overrides(gconfig.DEFAULT_CONFIG, config, overrides)
gconfig.validate_config(cfg) gconfig.validate_config(cfg, resume=resume is not None)
t = cfg["train"] t = cfg["train"]
_device = torch.device(device) if device else gconfig.auto_device() _device = torch.device(device) if device else gconfig.auto_device()
@@ -690,6 +734,7 @@ def train(
typer.echo(f"device: {_device}") typer.echo(f"device: {_device}")
typer.echo(f"out_dir: {out_dir}") typer.echo(f"out_dir: {out_dir}")
typer.echo(f"precision: {t['precision']}")
run_train_job( run_train_job(
data=data, data=data,
@@ -735,6 +780,10 @@ def new_run(
stage2_k_max: Annotated[Optional[int], typer.Option("--stage2-k-max")] = None, stage2_k_max: Annotated[Optional[int], typer.Option("--stage2-k-max")] = None,
stage2_context_dim: Annotated[Optional[int], typer.Option("--stage2-context-dim")] = None, stage2_context_dim: Annotated[Optional[int], typer.Option("--stage2-context-dim")] = None,
stage2_stage1_context: Annotated[Optional[Stage1Context], typer.Option("--stage2-stage1-context")] = None, stage2_stage1_context: Annotated[Optional[Stage1Context], typer.Option("--stage2-stage1-context")] = None,
stage1_init_from: Annotated[Optional[Path], typer.Option("--stage1-init-from")] = None,
stage1_freeze: Annotated[Optional[bool], typer.Option("--stage1-freeze/--no-stage1-freeze")] = None,
stage2_init_from: Annotated[Optional[Path], typer.Option("--stage2-init-from")] = None,
stage2_freeze: Annotated[Optional[bool], typer.Option("--stage2-freeze/--no-stage2-freeze")] = None,
conditioning: Annotated[Optional[Conditioning], typer.Option("--conditioning")] = None, conditioning: Annotated[Optional[Conditioning], typer.Option("--conditioning")] = None,
router: Annotated[Optional[bool], typer.Option("--router/--no-router")] = None, router: Annotated[Optional[bool], typer.Option("--router/--no-router")] = None,
router_type: Annotated[Optional[str], typer.Option("--router-type")] = None, router_type: Annotated[Optional[str], typer.Option("--router-type")] = None,
@@ -795,6 +844,10 @@ def new_run(
"stage2_k_max": stage2_k_max, "stage2_k_max": stage2_k_max,
"stage2_context_dim": stage2_context_dim, "stage2_context_dim": stage2_context_dim,
"stage2_stage1_context": stage2_stage1_context.value if stage2_stage1_context is not None else None, "stage2_stage1_context": stage2_stage1_context.value if stage2_stage1_context is not None else None,
"stage1_init_from": str(stage1_init_from) if stage1_init_from is not None else None,
"stage1_freeze": stage1_freeze,
"stage2_init_from": str(stage2_init_from) if stage2_init_from is not None else None,
"stage2_freeze": stage2_freeze,
"mode": mode.value if mode is not None else None, "mode": mode.value if mode is not None else None,
"stage1_generator": stage1_generator.value if stage1_generator is not None else None, "stage1_generator": stage1_generator.value if stage1_generator is not None else None,
"stage2_generator": stage2_generator.value if stage2_generator is not None else None, "stage2_generator": stage2_generator.value if stage2_generator is not None else None,
@@ -854,6 +907,52 @@ def new_run(
typer.echo(f" giant train {data_arg} --config {config_path} --out {run_dir}") typer.echo(f" giant train {data_arg} --config {config_path} --out {run_dir}")
model_app = typer.Typer(
no_args_is_help=True,
help="Inspect a resolved model architecture without training.",
)
app.add_typer(model_app, name="model")
@model_app.command("summary")
def model_summary(
config: Annotated[
Optional[Path],
typer.Option("--config", "-c", help="TOML config file (default: built-in defaults)"),
] = None,
pdg_vocab: Annotated[
int,
typer.Option(
"--pdg-vocab",
help="Placeholder PDG vocab size for conditioning.particle.type='embedding' "
"or a pdg/process router (no dataset attached to derive the real training vocab)",
),
] = 300,
mat_vocab: Annotated[
int,
typer.Option(
"--mat-vocab",
help="Placeholder material vocab size for conditioning.material.type='embedding' "
"or a process router (default: the number of known materials in giant.materials)",
),
] = len(MATERIAL_PROPERTIES),
) -> None:
"""Build the resolved model graph from a config with no dataset attached, and
print per-module parameter counts, trunk widths, which heads exist, and which
conditioning/stage1_model/stage2_model config keys actually shaped it."""
from giant.model.summary import render_summary, summarize_model
cfg = gconfig.merge_cli_overrides(gconfig.DEFAULT_CONFIG, config, {})
try:
gconfig.validate_config(cfg)
except ValueError as exc:
typer.echo(f"error: {exc}", err=True)
raise typer.Exit(1)
summary = summarize_model(cfg, pdg_vocab=pdg_vocab, mat_vocab=mat_vocab)
typer.echo(render_summary(summary))
@app.command() @app.command()
def predict( def predict(
data: Annotated[Path, typer.Argument(help="Parquet file or directory of parquet files")], data: Annotated[Path, typer.Argument(help="Parquet file or directory of parquet files")],
+145 -6
View File
@@ -468,6 +468,18 @@ class ParticleTypeConfig:
# silently the same number). 0 = inherit conditioning.particle.emb_dim, # silently the same number). 0 = inherit conditioning.particle.emb_dim,
# preserving pre-#29 behavior. # preserving pre-#29 behavior.
n_classes: int = 0 n_classes: int = 0
# Class-balances the target = "onehot" cross-entropy loss against the
# secondary-species long tail (gitea #44: the failure mode motivating the
# v0.3.0 pivot was specifically a species collapse — zero photon
# secondaries, hallucinated antineutrinos). "none": plain CE (pre-#44
# behavior). "inverse_freq": CE weighted by 1/count per class,
# normalized to mean 1 across classes so lambda_weight doesn't need
# retuning when this is switched on. validate_config requires target =
# "onehot" and stage2_model.generator != "wgan" whenever this isn't
# "none" — "embedding"/"physical" have no class CE to weight, and the
# WGAN stage-2 path feeds its type slice to the critic via a
# straight-through Gumbel relaxation instead of a CE loss.
class_weighting: str = "none"
@classmethod @classmethod
def from_dict(cls, d: dict | None) -> "ParticleTypeConfig": def from_dict(cls, d: dict | None) -> "ParticleTypeConfig":
@@ -477,6 +489,7 @@ class ParticleTypeConfig:
lambda_weight=d.get("lambda", 1.0), lambda_weight=d.get("lambda", 1.0),
other_policy=d.get("other_policy", "sample"), other_policy=d.get("other_policy", "sample"),
n_classes=d.get("n_classes", 0), n_classes=d.get("n_classes", 0),
class_weighting=d.get("class_weighting", "none"),
) )
def to_dict(self) -> dict: def to_dict(self) -> dict:
@@ -485,6 +498,7 @@ class ParticleTypeConfig:
"lambda": self.lambda_weight, "lambda": self.lambda_weight,
"other_policy": self.other_policy, "other_policy": self.other_policy,
"n_classes": self.n_classes, "n_classes": self.n_classes,
"class_weighting": self.class_weighting,
} }
@@ -594,6 +608,19 @@ class Stage1ModelConfig:
# false skips building/training stage 1 entirely. The resulting # false skips building/training stage 1 entirely. The resulting
# checkpoint holds only stage 2 and cannot be rolled out. # checkpoint holds only stage 2 and cannot be rolled out.
active: bool = True active: bool = True
# Checkpoint .pt to load this stage's weights from before training starts
# (its own "model"/"sec_decoder" key, not this run's own resume state) —
# "" means start from a fresh init. See `freeze` below for the partial-
# retrain use case this exists for (gitea #42).
init_from: str = ""
# true keeps this stage's weights exactly as loaded from `init_from` —
# forward/backward still run every batch (so its loss/grad_norm metrics
# stay meaningful, and a WGAN stage's critic still gets a real signal to
# report), but its optimizer never steps. Lets a rollout-capable
# checkpoint retrain only the *other* stage against a fixed, known-good
# one (gitea #42) — `validate_config` requires `init_from` to be set
# whenever this is true, unless the run is a `--resume`.
freeze: bool = False
# "flow": conditional flow matching (~10 ODE steps at inference). # "flow": conditional flow matching (~10 ODE steps at inference).
# "ddpm": cosine-schedule diffusion baseline. # "ddpm": cosine-schedule diffusion baseline.
# "wgan": WGAN-GP, single forward pass at inference. # "wgan": WGAN-GP, single forward pass at inference.
@@ -620,6 +647,8 @@ class Stage1ModelConfig:
d = d or {} d = d or {}
return cls( return cls(
active=d.get("active", True), active=d.get("active", True),
init_from=d.get("init_from", ""),
freeze=d.get("freeze", False),
generator=d.get("generator", "flow"), generator=d.get("generator", "flow"),
hidden_dim=d.get("hidden_dim", 256), hidden_dim=d.get("hidden_dim", 256),
n_res_blocks=d.get("n_res_blocks", 6), n_res_blocks=d.get("n_res_blocks", 6),
@@ -636,6 +665,8 @@ class Stage1ModelConfig:
def to_dict(self) -> dict: def to_dict(self) -> dict:
return { return {
"active": self.active, "active": self.active,
"init_from": self.init_from,
"freeze": self.freeze,
"generator": self.generator, "generator": self.generator,
"hidden_dim": self.hidden_dim, "hidden_dim": self.hidden_dim,
"n_res_blocks": self.n_res_blocks, "n_res_blocks": self.n_res_blocks,
@@ -655,6 +686,9 @@ class Stage2ModelConfig:
# false trains stage 1 alone. giant rollout must then refuse the # false trains stage 1 alone. giant rollout must then refuse the
# checkpoint; giant predict still works. # checkpoint; giant predict still works.
active: bool = True active: bool = True
# See Stage1ModelConfig.init_from/.freeze — same semantics, this stage.
init_from: str = ""
freeze: bool = False
# "one_shot": predict all k_max slots simultaneously with padded slots # "one_shot": predict all k_max slots simultaneously with padded slots
# masked from the loss (v0.2 behaviour). # masked from the loss (v0.2 behaviour).
# "autoregressive": emit one secondary at a time in descending-energy # "autoregressive": emit one secondary at a time in descending-energy
@@ -677,6 +711,13 @@ class Stage2ModelConfig:
# output, closing the train/inference gap at the cost of a sampling pass # output, closing the train/inference gap at the cost of a sampling pass
# per batch and a moving target early in training. # per batch and a moving target early in training.
stage1_context: str = "truth" stage1_context: str = "truth"
# Ramp for "sampled": P(condition on the ground-truth stage-1 outcome
# rather than a fresh sample), linearly interpolated from ctx_p_start
# (epoch 0) to ctx_p_end (the final epoch) — the same scheduled-sampling
# shape as autoregressive.tf_p_start/tf_p_end, so stage 2 doesn't chase a
# wildly moving stage-1 target in early epochs. Unread under "truth".
ctx_p_start: float = 1.0
ctx_p_end: float = 0.0
n_sec: NSecConfig = field(default_factory=NSecConfig) n_sec: NSecConfig = field(default_factory=NSecConfig)
particle_type: ParticleTypeConfig = field(default_factory=ParticleTypeConfig) particle_type: ParticleTypeConfig = field(default_factory=ParticleTypeConfig)
autoregressive: AutoregressiveConfig = field(default_factory=AutoregressiveConfig) autoregressive: AutoregressiveConfig = field(default_factory=AutoregressiveConfig)
@@ -692,6 +733,8 @@ class Stage2ModelConfig:
d = d or {} d = d or {}
return cls( return cls(
active=d.get("active", True), active=d.get("active", True),
init_from=d.get("init_from", ""),
freeze=d.get("freeze", False),
decoder=d.get("decoder", "autoregressive"), decoder=d.get("decoder", "autoregressive"),
generator=d.get("generator", "wgan"), generator=d.get("generator", "wgan"),
hidden_dim=d.get("hidden_dim", 256), hidden_dim=d.get("hidden_dim", 256),
@@ -701,6 +744,8 @@ class Stage2ModelConfig:
k_max=d.get("k_max", 15), k_max=d.get("k_max", 15),
context_dim=d.get("context_dim", 64), context_dim=d.get("context_dim", 64),
stage1_context=d.get("stage1_context", "truth"), stage1_context=d.get("stage1_context", "truth"),
ctx_p_start=d.get("ctx_p_start", 1.0),
ctx_p_end=d.get("ctx_p_end", 0.0),
n_sec=NSecConfig.from_dict(d.get("n_sec")), n_sec=NSecConfig.from_dict(d.get("n_sec")),
particle_type=ParticleTypeConfig.from_dict(d.get("particle_type")), particle_type=ParticleTypeConfig.from_dict(d.get("particle_type")),
autoregressive=AutoregressiveConfig.from_dict(d.get("autoregressive")), autoregressive=AutoregressiveConfig.from_dict(d.get("autoregressive")),
@@ -715,6 +760,8 @@ class Stage2ModelConfig:
def to_dict(self) -> dict: def to_dict(self) -> dict:
return { return {
"active": self.active, "active": self.active,
"init_from": self.init_from,
"freeze": self.freeze,
"decoder": self.decoder, "decoder": self.decoder,
"generator": self.generator, "generator": self.generator,
"hidden_dim": self.hidden_dim, "hidden_dim": self.hidden_dim,
@@ -724,6 +771,8 @@ class Stage2ModelConfig:
"k_max": self.k_max, "k_max": self.k_max,
"context_dim": self.context_dim, "context_dim": self.context_dim,
"stage1_context": self.stage1_context, "stage1_context": self.stage1_context,
"ctx_p_start": self.ctx_p_start,
"ctx_p_end": self.ctx_p_end,
"n_sec": self.n_sec.to_dict(), "n_sec": self.n_sec.to_dict(),
"particle_type": self.particle_type.to_dict(), "particle_type": self.particle_type.to_dict(),
"autoregressive": self.autoregressive.to_dict(), "autoregressive": self.autoregressive.to_dict(),
@@ -766,6 +815,12 @@ class TrainConfig:
# thousands of steps. Per-epoch metrics (the metrics.csv row) always log # thousands of steps. Per-epoch metrics (the metrics.csv row) always log
# in full. # in full.
wandb_log_every: int = 50 wandb_log_every: int = 50
# Training-step autocast dtype: "fp32" (default, no autocast) or "bf16".
# No "fp16" — GradScaler and the double-backward in
# giant.model.wgan.gradient_penalty don't mix well, and bf16 alone covers
# every training GPU in the fleet (Ampere and newer). See
# giant.training.amp.resolve_autocast (gitea #47).
precision: str = "fp32"
@classmethod @classmethod
def from_dict(cls, d: dict | None) -> "TrainConfig": def from_dict(cls, d: dict | None) -> "TrainConfig":
@@ -787,6 +842,7 @@ class TrainConfig:
wandb_project=d.get("wandb_project", "giant"), wandb_project=d.get("wandb_project", "giant"),
wandb_run_name=d.get("wandb_run_name", ""), wandb_run_name=d.get("wandb_run_name", ""),
wandb_log_every=d.get("wandb_log_every", 50), wandb_log_every=d.get("wandb_log_every", 50),
precision=d.get("precision", "fp32"),
) )
def to_dict(self) -> dict: def to_dict(self) -> dict:
@@ -807,6 +863,7 @@ class TrainConfig:
"wandb_project": self.wandb_project, "wandb_project": self.wandb_project,
"wandb_run_name": self.wandb_run_name, "wandb_run_name": self.wandb_run_name,
"wandb_log_every": self.wandb_log_every, "wandb_log_every": self.wandb_log_every,
"precision": self.precision,
} }
@@ -843,6 +900,25 @@ class GiantConfig:
DEFAULT_CONFIG: dict = GiantConfig().to_dict() DEFAULT_CONFIG: dict = GiantConfig().to_dict()
def leaf_paths(node: dict, prefix: str = "") -> list[str]:
"""Every dotted leaf path in a DEFAULT_CONFIG-shaped dict, e.g.
"stage1_model.router.n_experts". `[meta]` (run provenance, no schema
counterpart) is skipped at the top level, matching `validate_config_keys`.
Shared by `tests/test_config_consumed_keys.py` (the static per-identifier
audit) and `giant.model.summary` (the runtime per-config audit, gitea
#46) so both walk the exact same tree."""
paths = []
for key, value in node.items():
if prefix == "" and key == "meta":
continue
path = f"{prefix}.{key}" if prefix else key
if isinstance(value, dict):
paths.extend(leaf_paths(value, path))
else:
paths.append(path)
return paths
def git_hash() -> str: def git_hash() -> str:
try: try:
return subprocess.check_output(["git", "rev-parse", "HEAD"], stderr=subprocess.DEVNULL).decode().strip() return subprocess.check_output(["git", "rev-parse", "HEAD"], stderr=subprocess.DEVNULL).decode().strip()
@@ -1060,6 +1136,7 @@ FLAG_SPECS: tuple[FlagSpec, ...] = (
FlagSpec("wandb_project", ("train.wandb_project",)), FlagSpec("wandb_project", ("train.wandb_project",)),
FlagSpec("wandb_run_name", ("train.wandb_run_name",)), FlagSpec("wandb_run_name", ("train.wandb_run_name",)),
FlagSpec("wandb_log_every", ("train.wandb_log_every",)), FlagSpec("wandb_log_every", ("train.wandb_log_every",)),
FlagSpec("precision", ("train.precision",)),
# --hidden-dim/--n-blocks/--dropout are stage-1-only backward-compat # --hidden-dim/--n-blocks/--dropout are stage-1-only backward-compat
# shorthands (they predate stage2_model having its own flags); # shorthands (they predate stage2_model having its own flags);
# --stage1-* wins when both are given. # --stage1-* wins when both are given.
@@ -1112,6 +1189,13 @@ FLAG_SPECS: tuple[FlagSpec, ...] = (
FlagSpec("stage1_critic_n_res_blocks", ("stage1_model.wgan.critic_n_res_blocks",)), FlagSpec("stage1_critic_n_res_blocks", ("stage1_model.wgan.critic_n_res_blocks",)),
FlagSpec("stage2_critic_hidden_dim", ("stage2_model.wgan.critic_hidden_dim",)), FlagSpec("stage2_critic_hidden_dim", ("stage2_model.wgan.critic_hidden_dim",)),
FlagSpec("stage2_critic_n_res_blocks", ("stage2_model.wgan.critic_n_res_blocks",)), FlagSpec("stage2_critic_n_res_blocks", ("stage2_model.wgan.critic_n_res_blocks",)),
# Partial-retrain (gitea #42): stage-scoped only, no shared alias — a
# shared "freeze both stages from the same file" flag has no sensible
# meaning (a checkpoint has one set of weights per stage).
FlagSpec("stage1_init_from", ("stage1_model.init_from",)),
FlagSpec("stage1_freeze", ("stage1_model.freeze",)),
FlagSpec("stage2_init_from", ("stage2_model.init_from",)),
FlagSpec("stage2_freeze", ("stage2_model.freeze",)),
) )
@@ -1344,7 +1428,7 @@ def merge_cli_overrides(
return cfg return cfg
def validate_config(cfg: dict) -> None: def validate_config(cfg: dict, *, resume: bool = False) -> None:
"""Cross-block validation the per-block schema can't express on its own. """Cross-block validation the per-block schema can't express on its own.
Raises ValueError with a clear message on the first violation found. Call Raises ValueError with a clear message on the first violation found. Call
@@ -1352,6 +1436,10 @@ def validate_config(cfg: dict) -> None:
these checks need to see across blocks, so they don't belong in these checks need to see across blocks, so they don't belong in
`migrate_config` (which only ever sees one dict's own keys) or in any `migrate_config` (which only ever sees one dict's own keys) or in any
single block's defaults. single block's defaults.
`resume=True` (only `giant train --resume` passes this) relaxes the
`stage{1,2}_model.freeze` -> `.init_from` requirement below: a resumed
frozen stage's weights come from the resume checkpoint, not `init_from`.
""" """
particle_type = _get_path(cfg, "conditioning.particle.type") particle_type = _get_path(cfg, "conditioning.particle.type")
@@ -1364,7 +1452,34 @@ def validate_config(cfg: dict) -> None:
f"{particle_type!r})" f"{particle_type!r})"
) )
class_weighting = _get_path(cfg, "stage2_model.particle_type.class_weighting")
if class_weighting not in ("none", "inverse_freq"):
raise ValueError(
f"stage2_model.particle_type.class_weighting = {class_weighting!r} — must be 'none' or 'inverse_freq'"
)
if class_weighting != "none" and pt_target != "onehot":
raise ValueError(
"stage2_model.particle_type.class_weighting != 'none' requires "
f"stage2_model.particle_type.target = 'onehot' (there is no class "
f"cross-entropy to weight under target = {pt_target!r})"
)
if class_weighting != "none" and _get_path(cfg, "stage2_model.generator") == "wgan":
raise ValueError(
"stage2_model.particle_type.class_weighting != 'none' is "
"incompatible with stage2_model.generator = 'wgan' — that path "
"feeds the type slice to the critic via a straight-through "
"Gumbel relaxation instead of a class cross-entropy, so there is "
"nothing to weight"
)
for stage_name in ("stage1_model", "stage2_model"): for stage_name in ("stage1_model", "stage2_model"):
if _get_path(cfg, f"{stage_name}.freeze") and not _get_path(cfg, f"{stage_name}.init_from") and not resume:
raise ValueError(
f"{stage_name}.freeze = true requires {stage_name}.init_from "
"to be set (or --resume) — freezing a randomly-initialized "
"model is almost certainly a mistake"
)
router = _get_path(cfg, f"{stage_name}.router") or {} router = _get_path(cfg, f"{stage_name}.router") or {}
if router.get("enabled") and router.get("type") in ("pdg", "process") and particle_type == "physical": if router.get("enabled") and router.get("type") in ("pdg", "process") and particle_type == "physical":
raise ValueError( raise ValueError(
@@ -1401,14 +1516,36 @@ def validate_config(cfg: dict) -> None:
if stop_sampling not in ("greedy", "sample"): if stop_sampling not in ("greedy", "sample"):
raise ValueError(f"stage2_model.n_sec.stop_sampling = {stop_sampling!r} — must be 'greedy' or 'sample'") raise ValueError(f"stage2_model.n_sec.stop_sampling = {stop_sampling!r} — must be 'greedy' or 'sample'")
if _get_path(cfg, "stage2_model.stage1_context") == "sampled": precision = _get_path(cfg, "train.precision")
if precision not in ("fp32", "bf16"):
raise ValueError( raise ValueError(
"stage2_model.stage1_context = 'sampled' is accepted by the schema " f"train.precision = {precision!r} — must be 'fp32' or 'bf16' "
"but not implemented — trainers.py always trains stage 2 against " "('fp16' is not supported: see giant.training.amp.resolve_autocast)"
"the ground-truth stage-1 output; use 'truth' (default) instead "
"(see issues.md Issue 16 for the planned implementation)"
) )
stage1_context = _get_path(cfg, "stage2_model.stage1_context")
if stage1_context not in ("truth", "sampled"):
raise ValueError(f"stage2_model.stage1_context = {stage1_context!r} — must be 'truth' or 'sampled'")
if stage1_context == "sampled":
if not (_get_path(cfg, "stage1_model.active") and _get_path(cfg, "stage2_model.active")):
raise ValueError(
"stage2_model.stage1_context = 'sampled' requires both "
"stage1_model.active and stage2_model.active = true — there is "
"no stage-1 model to sample from in a stage-2-only run"
)
ctx_p_start = _get_path(cfg, "stage2_model.ctx_p_start")
ctx_p_end = _get_path(cfg, "stage2_model.ctx_p_end")
for name, value in (("ctx_p_start", ctx_p_start), ("ctx_p_end", ctx_p_end)):
if not (0.0 <= value <= 1.0):
raise ValueError(f"stage2_model.{name} = {value} — must be in [0, 1]")
if ctx_p_start == 1.0 and ctx_p_end == 1.0:
raise ValueError(
"stage2_model.stage1_context = 'sampled' with ctx_p_start = "
"ctx_p_end = 1.0 always conditions on the ground truth — "
"identical to 'truth' but silently so; use 'truth' instead or "
"lower ctx_p_end"
)
if ( if (
_get_path(cfg, "stage2_model.n_sec.mode") == "truth" _get_path(cfg, "stage2_model.n_sec.mode") == "truth"
and _get_path(cfg, "stage1_model.active") and _get_path(cfg, "stage1_model.active")
@@ -1569,6 +1706,8 @@ _OUT_DIR_NAME_CANDIDATES = [
), ),
("particle_conditioning", _conditioning_candidate("particle", "c")), ("particle_conditioning", _conditioning_candidate("particle", "c")),
("material_conditioning", _conditioning_candidate("material", "m")), ("material_conditioning", _conditioning_candidate("material", "m")),
("stage1_freeze", _path_candidate("stage1_model.freeze", "s1frozen", formatter=lambda _: "")),
("stage2_freeze", _path_candidate("stage2_model.freeze", "s2frozen", formatter=lambda _: "")),
("stage1_hidden_dim", _path_candidate("stage1_model.hidden_dim", "h")), ("stage1_hidden_dim", _path_candidate("stage1_model.hidden_dim", "h")),
("stage2_hidden_dim", _path_candidate("stage2_model.hidden_dim", "s2h")), ("stage2_hidden_dim", _path_candidate("stage2_model.hidden_dim", "s2h")),
("stage1_n_res_blocks", _path_candidate("stage1_model.n_res_blocks", "b")), ("stage1_n_res_blocks", _path_candidate("stage1_model.n_res_blocks", "b")),
+24 -11
View File
@@ -1,4 +1,4 @@
from dataclasses import dataclass from dataclasses import dataclass, field
from pathlib import Path from pathlib import Path
from typing import Iterator from typing import Iterator
@@ -256,24 +256,32 @@ def _rank_by_frequency_from_files(files: list[Path], column: str, cast) -> dict:
return counts return counts
def _topn_plus_other_map(counts: dict, n_classes: int) -> tuple[dict, dict]: def _topn_plus_other_map(counts: dict, n_classes: int) -> tuple[dict, dict, dict]:
"""Frequency-capped value->index map: the `n_classes - 1` most frequent """Frequency-capped value->index map: the `n_classes - 1` most frequent
keys get their own index; every rarer key is bucketed into a shared keys get their own index; every rarer key is bucketed into a shared
"other" index (`n_classes - 1`). "other" index (`n_classes - 1`).
Returns `(class_map, other_members)` `other_members` is `{key: count}` Returns `(class_map, other_members, class_counts)` `other_members` is
for every key bucketed into "other" (the empirical within-bucket `{key: count}` for every key bucketed into "other" (the empirical
distribution, for `other_policy = "sample"` at rollout). within-bucket distribution, for `other_policy = "sample"` at rollout);
`class_counts` is `{index: total_count}` for every resulting class index
(0-indexed; the "other" index's count is the sum of `other_members`),
the per-class frequencies `stage2_model.particle_type.class_weighting`
(gitea #44) needs and that would otherwise be dropped once `counts` is
collapsed into `class_map`.
""" """
ranked = sorted(counts, key=lambda k: counts[k], reverse=True) ranked = sorted(counts, key=lambda k: counts[k], reverse=True)
keep = ranked[: max(n_classes - 1, 0)] keep = ranked[: max(n_classes - 1, 0)]
class_map = {k: i for i, k in enumerate(keep)} class_map = {k: i for i, k in enumerate(keep)}
class_counts = {i: counts[k] for i, k in enumerate(keep)}
other_idx = n_classes - 1 other_idx = n_classes - 1
other_members: dict = {} other_members: dict = {}
for k in ranked[len(keep) :]: for k in ranked[len(keep) :]:
class_map[k] = other_idx class_map[k] = other_idx
other_members[k] = counts[k] other_members[k] = counts[k]
return class_map, other_members if other_members:
class_counts[other_idx] = sum(other_members.values())
return class_map, other_members, class_counts
def build_process_map_from_files(files: list[Path], n_experts: int) -> dict[str, int]: def build_process_map_from_files(files: list[Path], n_experts: int) -> dict[str, int]:
@@ -287,7 +295,7 @@ def build_process_map_from_files(files: list[Path], n_experts: int) -> dict[str,
fixed-width n_sec_head classifier. fixed-width n_sec_head classifier.
""" """
counts = _rank_by_frequency_from_files(files, "process", str) counts = _rank_by_frequency_from_files(files, "process", str)
class_map, _ = _topn_plus_other_map(counts, n_experts) class_map, _, _ = _topn_plus_other_map(counts, n_experts)
return class_map return class_map
@@ -299,6 +307,11 @@ class TopNMap:
class_map: dict class_map: dict
other_members: dict other_members: dict
# {class_index: total_count} — see _topn_plus_other_map. Empty for a
# TopNMap decoded from a checkpoint/sidecar predating gitea #44; only
# stage2_model.particle_type.class_weighting reads it, and it raises
# loudly if it needs counts that aren't there (giant/training/trainers.py).
class_counts: dict = field(default_factory=dict)
def build_topn_map_from_files(files: list[Path], column: str, n_classes: int, cast=str) -> TopNMap: def build_topn_map_from_files(files: list[Path], column: str, n_classes: int, cast=str) -> TopNMap:
@@ -315,8 +328,8 @@ def build_topn_map_from_files(files: list[Path], column: str, n_classes: int, ca
free during this same scan. free during this same scan.
""" """
counts = _rank_by_frequency_from_files(files, column, cast) counts = _rank_by_frequency_from_files(files, column, cast)
class_map, other_members = _topn_plus_other_map(counts, n_classes) class_map, other_members, class_counts = _topn_plus_other_map(counts, n_classes)
return TopNMap(class_map=class_map, other_members=other_members) return TopNMap(class_map=class_map, other_members=other_members, class_counts=class_counts)
def build_pdg_topn_map_from_files(files: list[Path], n_classes: int) -> TopNMap: def build_pdg_topn_map_from_files(files: list[Path], n_classes: int) -> TopNMap:
@@ -347,5 +360,5 @@ def build_pdg_topn_map_from_files(files: list[Path], n_classes: int) -> TopNMap:
if has_sec: if has_sec:
exploded = df["sec_pdg_list"].explode().dropna() exploded = df["sec_pdg_list"].explode().dropna()
_accumulate_value_counts(counts, exploded, int) _accumulate_value_counts(counts, exploded, int)
class_map, other_members = _topn_plus_other_map(counts, n_classes) class_map, other_members, class_counts = _topn_plus_other_map(counts, n_classes)
return TopNMap(class_map=class_map, other_members=other_members) return TopNMap(class_map=class_map, other_members=other_members, class_counts=class_counts)
+10 -1
View File
@@ -35,7 +35,10 @@ from giant.data.transforms import Normalizer, sorted_membership
# v3: NormalizerEntry.energy_reservoir_sample (100k raw values) replaced by # v3: NormalizerEntry.energy_reservoir_sample (100k raw values) replaced by
# energy_quantiles (a fixed ENERGY_QUANTILE_LEVELS-point quantile grid) — a # energy_quantiles (a fixed ENERGY_QUANTILE_LEVELS-point quantile grid) — a
# v2 sidecar has no such grid to fall back on, so it must be recomputed. # v2 sidecar has no such grid to fall back on, so it must be recomputed.
_CACHE_FORMAT_VERSION = 3 # v4: TopNMap gained class_counts (gitea #44, stage2_model.particle_type.
# class_weighting) — a v3 sidecar's cached topn_maps have no counts, so they
# must be rebuilt rather than silently cached with class_counts={}.
_CACHE_FORMAT_VERSION = 4
_DIMS = { _DIMS = {
"COND_DIM": COND_DIM, "COND_DIM": COND_DIM,
@@ -131,6 +134,7 @@ def topnmap_to_json(m: TopNMap) -> dict:
return { return {
"class_map": {str(k): v for k, v in m.class_map.items()}, "class_map": {str(k): v for k, v in m.class_map.items()},
"other_members": {str(k): v for k, v in m.other_members.items()}, "other_members": {str(k): v for k, v in m.other_members.items()},
"class_counts": {str(k): v for k, v in m.class_counts.items()},
} }
@@ -139,6 +143,11 @@ def topnmap_from_json(d: dict, axis: str) -> TopNMap:
return TopNMap( return TopNMap(
class_map={cast(k): v for k, v in d["class_map"].items()}, class_map={cast(k): v for k, v in d["class_map"].items()},
other_members={cast(k): v for k, v in d["other_members"].items()}, other_members={cast(k): v for k, v in d["other_members"].items()},
# Missing for a checkpoint's topn maps predating gitea #44 — {} is
# the correct decode there (inference never reads class_counts; only
# stage2_model.particle_type.class_weighting does, at train time, and
# it raises loudly if it needs counts a checkpoint doesn't have).
class_counts={int(k): v for k, v in d.get("class_counts", {}).items()},
) )
+17
View File
@@ -63,6 +63,23 @@ def build_history(name: str, in_dim: int, out_dim: int, **kwargs) -> HistoryEnco
return cls(in_dim, out_dim, **filtered) return cls(in_dim, out_dim, **filtered)
@register_history("none")
class NoHistory(HistoryEncoder):
"""No history signal at all — ignores feat/has_prev entirely and always
returns zeros. Ablates whether the AR decoder's history conditioning is
earning its parameters. `init_cache`/`step` use the base class's O(1)
defaults unmodified (this encoder's own `forward` is already O(1) per
call regardless of prefix length)."""
def __init__(self, in_dim: int, out_dim: int) -> None:
super().__init__()
self.out_dim = out_dim
def forward(self, feat: torch.Tensor, has_prev: torch.Tensor) -> torch.Tensor:
B, K, _ = feat.shape
return torch.zeros(B, K, self.out_dim, device=feat.device, dtype=feat.dtype)
@register_history("markov") @register_history("markov")
class MarkovHistory(HistoryEncoder): class MarkovHistory(HistoryEncoder):
"""Summarizes the previous secondary's own `(energy_fraction, direction, """Summarizes the previous secondary's own `(energy_fraction, direction,
+6
View File
@@ -15,6 +15,7 @@ from giant.model.history import (
AttentionHistory, AttentionHistory,
HistoryEncoder, HistoryEncoder,
MarkovHistory, MarkovHistory,
NoHistory,
_CausalAttnBlock, _CausalAttnBlock,
build_history, build_history,
register_history, register_history,
@@ -54,6 +55,7 @@ from giant.model.routers import (
ROUTER_REGISTRY, ROUTER_REGISTRY,
ComposedRouter, ComposedRouter,
EnergyRouter, EnergyRouter,
NoneRouter,
PdgRouter, PdgRouter,
ProcessRouter, ProcessRouter,
Router, Router,
@@ -67,6 +69,7 @@ from giant.model.routers import (
from giant.model.trunks import ( from giant.model.trunks import (
TRUNK_REGISTRY, TRUNK_REGISTRY,
ExpertTrunk, ExpertTrunk,
LinearTrunk,
RoutedTrunk, RoutedTrunk,
Trunk, Trunk,
_route_forward, _route_forward,
@@ -90,7 +93,10 @@ __all__ = [
"FlowObjective", "FlowObjective",
"HISTORY_REGISTRY", "HISTORY_REGISTRY",
"HistoryEncoder", "HistoryEncoder",
"LinearTrunk",
"MarkovHistory", "MarkovHistory",
"NoHistory",
"NoneRouter",
"OBJECTIVE_REGISTRY", "OBJECTIVE_REGISTRY",
"Objective", "Objective",
"PdgRouter", "PdgRouter",
+53 -15
View File
@@ -45,21 +45,36 @@ class Router(nn.Module):
straight-through Gumbel-softmax (`gumbel=True`, train mode only): straight-through Gumbel-softmax (`gumbel=True`, train mode only):
hardens the forward pass to a one-hot sample (matching eval-time hardens the forward pass to a one-hot sample (matching eval-time
top-1 dispatch) while keeping the soft sample's gradient on backward. top-1 dispatch) while keeping the soft sample's gradient on backward.
Forced fp32 (`torch.autocast(..., enabled=False)`) regardless of the
caller's ambient `train.precision` autocast region: `clamp_min(1e-8)`
below sits under bf16's precision but *above* fp16's ~6e-8 subnormal
floor, so `log_probs` degrading here is exactly the kind of quiet
drift that cost a whole rollout benchmark before (see the MoE section
of CLAUDE.md's Roadmap) — cheap to rule out (gitea #47).
""" """
probs = self.gate(cond_cont, cond_cat) with torch.autocast(cond_cont.device.type, enabled=False):
if not (self.gumbel and self.training): probs = self.gate(cond_cont, cond_cat)
return probs if not (self.gumbel and self.training):
log_probs = torch.log(probs.clamp_min(1e-8)) return probs
return F.gumbel_softmax(log_probs, tau=self.gumbel_tau, hard=True, dim=-1) log_probs = torch.log(probs.clamp_min(1e-8))
return F.gumbel_softmax(log_probs, tau=self.gumbel_tau, hard=True, dim=-1)
def top1(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor) -> torch.Tensor: def top1(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor) -> torch.Tensor:
"""(B,) hard expert index, used for eval-time grouped dispatch.""" """(B,) hard expert index, used for eval-time grouped dispatch."""
return self.gate(cond_cont, cond_cat).argmax(dim=-1) with torch.autocast(cond_cont.device.type, enabled=False):
return self.gate(cond_cont, cond_cat).argmax(dim=-1)
def balance_loss(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor) -> torch.Tensor: def balance_loss(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor) -> torch.Tensor:
"""Importance CV^2 load-balancing auxiliary loss (Shazeer et al. 2017).""" """Importance CV^2 load-balancing auxiliary loss (Shazeer et al. 2017).
importance = self.gate(cond_cont, cond_cat).sum(dim=0) # (n_experts,)
return (importance.std() / (importance.mean() + 1e-8)) ** 2 Forced fp32 `importance` sums `gate()` over the whole batch (a
large-magnitude accumulation in reduced precision), then takes a
`std/mean` ratio: a classic catastrophic-cancellation shape (gitea
#47)."""
with torch.autocast(cond_cont.device.type, enabled=False):
importance = self.gate(cond_cont, cond_cat).sum(dim=0) # (n_experts,)
return (importance.std() / (importance.mean() + 1e-8)) ** 2
def classify_loss(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor, labels: torch.Tensor) -> torch.Tensor: def classify_loss(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor, labels: torch.Tensor) -> torch.Tensor:
"""Optional supervised auxiliary loss shaping the router's own belief. """Optional supervised auxiliary loss shaping the router's own belief.
@@ -76,12 +91,17 @@ class Router(nn.Module):
def gate_stats(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]: def gate_stats(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
"""Diagnostics: `(norm_entropy, importance)` — see v0.2 docstring for """Diagnostics: `(norm_entropy, importance)` — see v0.2 docstring for
the full explanation, unchanged in v0.3.0.""" the full explanation, unchanged in v0.3.0.
gate = self.gate(cond_cont, cond_cat) # (B, n_experts)
row_entropy = -(gate * (gate + 1e-8).log()).sum(dim=-1) # (B,) Forced fp32, same rationale as `balance_loss`/`combine_weights`: the
norm_entropy = row_entropy.mean() / math.log(self.n_experts) `+ 1e-8` epsilon here is `entropy_loss`'s training-loss path too, not
importance = gate.sum(dim=0) # (n_experts,) just a diagnostic (gitea #47)."""
return norm_entropy, importance with torch.autocast(cond_cont.device.type, enabled=False):
gate = self.gate(cond_cont, cond_cat) # (B, n_experts)
row_entropy = -(gate * (gate + 1e-8).log()).sum(dim=-1) # (B,)
norm_entropy = row_entropy.mean() / math.log(self.n_experts)
importance = gate.sum(dim=0) # (n_experts,)
return norm_entropy, importance
ROUTER_REGISTRY: dict[str, type[Router]] = {} ROUTER_REGISTRY: dict[str, type[Router]] = {}
@@ -124,6 +144,24 @@ def _inverse_bounded_interp(value: float, lo: float, hi: float) -> float:
return math.log(p / (1 - p)) return math.log(p / (1 - p))
@register_router("none")
class NoneRouter(Router):
"""Uniform 1/n_experts gate — no learned routing signal at all.
Still builds n_experts expert trunks via RoutedTrunk (same parameter
budget as a real router), but every row gets an identical weight
regardless of conditioning. Ablates whether the *learned routing
signal* as opposed to simply having multiple experts is earning
its parameters. `top1()` (the base class default) always dispatches to
expert 0 (argmax of a uniform vector), which still exercises
RoutedTrunk's real per-expert grouped-dispatch code path at eval time.
"""
def gate(self, cond_cont: torch.Tensor, cond_cat: torch.Tensor) -> torch.Tensor:
B = cond_cont.shape[0]
return torch.full((B, self.n_experts), 1.0 / self.n_experts, device=cond_cont.device)
@register_router("energy") @register_router("energy")
class EnergyRouter(Router): class EnergyRouter(Router):
"""Soft turn-on gate over normalized pre-step log-energy. """Soft turn-on gate over normalized pre-step log-energy.
+318
View File
@@ -0,0 +1,318 @@
"""Build-only model introspection (gitea #46): construct the resolved
Stage1/Stage2/critic graph from a config with no dataset attached, and report
per-module parameter counts, trunk widths, which heads exist, and via
differential probing which `conditioning`/`stage1_model`/`stage2_model`
config keys actually shape the built model. This is the runtime counterpart
to `tests/test_config_consumed_keys.py`'s static per-identifier audit: that
test asks "does any code reference this key's name at all", this module asks
"given *this* resolved config, does the key change what `build_models`/
`build_critics` (`giant/model/builders.py`) actually produces".
Differential probing, not identifier matching: build the model once from the
resolved config and take a structural fingerprint (`_fingerprint` which
submodules exist, every parameter's/buffer's shape+dtype, every plain scalar
attribute stored on any module). Then, for each in-scope leaf key, perturb
just that one value (`_perturb`), rebuild, and re-fingerprint. A changed
fingerprint or a rebuild that raises means the key was consumed; an
identical fingerprint means construction never looked at it under this
particular config. A key can be genuinely inert under one config and live
under another (e.g. any `stage1_model.router.*` key when `router.enabled =
false`) that config-dependence is exactly the "silently degenerate
combination" issue #46 is after, so it is reported per-run rather than
baked into a static table.
Keys legitimately owned by the trainer/sampler/rollout rather than by
`build_models`/`build_critics` (loss weights, WGAN-GP training
hyperparameters, teacher-forcing and stage1-context schedules, ...) are
cataloged in `_NOT_BUILD_TIME` below so the report doesn't flag them as
suspicious. One leaf is inert under every config today
`stage2_model.autoregressive.order` matching
`tests/test_config_consumed_keys.py`'s own `_KNOWN_UNUSED` entry; it is
deliberately *not* in `_NOT_BUILD_TIME`, since "always inert" is itself the
finding those two tests independently converge on.
"""
import copy
from dataclasses import dataclass, field
import torch.nn as nn
from giant.config import _get_path, _set_path, leaf_paths
from giant.model.builders import build_critics, build_models
from giant.model.trunks import RoutedTrunk
_IN_SCOPE_ROOTS = ("conditioning", "stage1_model", "stage2_model")
_PROBE_STR = "__giant_model_summary_probe__"
# A handful of string leaves branch on equality against one specific literal
# (e.g. `builders.py`: `stop_token = s2_spec.n_sec.mode == "stop_token"`),
# where every value other than that literal behaves identically. A single
# generic sentinel probe would then falsely read as inert whenever the
# config's *current* value is already one of those identically-behaving
# "other" values (e.g. mode="head") — it never crosses the one boundary that
# actually matters. Named here so probing tries the real alternative(s) too;
# every other string leaf is registry-validated (raises on garbage, still
# correctly detected as consumed) or genuinely value-independent, so doesn't
# need an entry.
_STRING_ALTERNATIVES: dict[str, tuple[str, ...]] = {
"stage2_model.n_sec.owner": ("stage1", "stage2"),
"stage2_model.n_sec.mode": ("stop_token", "head", "truth"),
"stage2_model.particle_type.target": ("physical", "onehot", "embedding"),
}
# Verified by reading giant/training/trainers.py, giant/training/stage2_inputs.py
# and giant/rollout.py while implementing gitea #46 — not auto-derived, so a
# future reader touching these fields should re-check this table still holds.
_NOT_BUILD_TIME: dict[str, str] = {
"stage1_model.init_from": "training/checkpoint.py's init_stages_from_checkpoints, run before build_stage_trainers (gitea #42)",
"stage1_model.freeze": "trainers.py: StageSpec.freeze, gates StageTrainer._step_optimizer (gitea #42)",
"stage2_model.init_from": "training/checkpoint.py's init_stages_from_checkpoints, run before build_stage_trainers (gitea #42)",
"stage2_model.freeze": "trainers.py: StageSpec.freeze, gates StageTrainer._step_optimizer (gitea #42)",
"stage1_model.lambda": "trainers.py: StageSpec.lambda_weight, the total-loss mix weight",
"stage2_model.lambda": "trainers.py: StageSpec.lambda_weight, the total-loss mix weight",
"stage2_model.n_sec.lambda": "trainers.py: StageSpec.n_sec_lambda, the n_sec-head loss weight",
"stage2_model.particle_type.lambda": "trainers.py: Stage2Trainer.particle_type_lambda, the type-head loss weight",
"stage2_model.particle_type.other_policy": "giant/rollout.py: resolves an 'other'-bucket secondary's PDG code at inference",
"stage2_model.particle_type.class_weighting": "trainers.py: FlowDDPMStageTrainer.type_class_weights, shapes the type-head loss, not the built graph (gitea #44)",
"stage2_model.autoregressive.teacher_forcing": "giant/training/stage2_inputs.py's training-time input assembly",
"stage2_model.autoregressive.tf_p_start": "trainers.py's teacher-forcing schedule",
"stage2_model.autoregressive.tf_p_end": "trainers.py's teacher-forcing schedule",
"stage2_model.stage1_context": "trainers.py's stage1/stage2 boundary — StageTrainer._stage1_context",
"stage2_model.ctx_p_start": "trainers.py's stage1-context sampling schedule",
"stage2_model.ctx_p_end": "trainers.py's stage1-context sampling schedule",
"stage1_model.router.lambda_balance": "trainers.py's load-balancing auxiliary loss weight",
"stage1_model.router.lambda_entropy": "trainers.py's entropy-regularization auxiliary loss weight",
"stage1_model.router.lambda_proc": "trainers.py's supervised process-classification auxiliary loss weight",
"stage1_model.router.gumbel_tau_start": "trainers.py's expert-combination Gumbel-softmax temperature anneal",
"stage1_model.router.gumbel_tau_end": "trainers.py's expert-combination Gumbel-softmax temperature anneal",
"stage2_model.router.lambda_balance": "trainers.py's load-balancing auxiliary loss weight",
"stage2_model.router.lambda_entropy": "trainers.py's entropy-regularization auxiliary loss weight",
"stage2_model.router.lambda_proc": "trainers.py's supervised process-classification auxiliary loss weight",
"stage2_model.router.gumbel_tau_start": "trainers.py's expert-combination Gumbel-softmax temperature anneal",
"stage2_model.router.gumbel_tau_end": "trainers.py's expert-combination Gumbel-softmax temperature anneal",
"stage1_model.wgan.n_critic": "trainers.py's WGAN-GP critic-update cadence",
"stage1_model.wgan.gp_weight": "trainers.py's WGAN-GP gradient-penalty coefficient",
"stage1_model.wgan.critic_lr": "trainers.py's critic optimizer learning rate",
"stage2_model.wgan.n_critic": "trainers.py's WGAN-GP critic-update cadence",
"stage2_model.wgan.gp_weight": "trainers.py's WGAN-GP gradient-penalty coefficient",
"stage2_model.wgan.critic_lr": "trainers.py's critic optimizer learning rate",
"stage2_model.wgan.gumbel_tau_start": "trainers.py's type-slice Gumbel-softmax temperature anneal (type_gumbel_tau_start)",
"stage2_model.wgan.gumbel_tau_end": "trainers.py's type-slice Gumbel-softmax temperature anneal (type_gumbel_tau_end)",
}
@dataclass
class ModelSummary:
modules: dict[str, nn.Module]
consumed: list[str]
inert: list[str]
elsewhere: list[str]
pdg_vocab: int
mat_vocab: int
vocab_caveats: list[str] = field(default_factory=list)
def _build_model_config(cfg: dict, pdg_vocab: int, mat_vocab: int) -> dict:
return {
"pdg_vocab": pdg_vocab,
"mat_vocab": mat_vocab,
"conditioning": cfg["conditioning"],
"stage1_model": cfg["stage1_model"],
"stage2_model": cfg["stage2_model"],
}
def _built_modules(cfg: dict, pdg_vocab: int, mat_vocab: int) -> dict[str, nn.Module]:
model_config = _build_model_config(cfg, pdg_vocab, mat_vocab)
modules: dict[str, nn.Module] = {}
for name, m in build_models(model_config).items():
if m is not None:
modules[name] = m
for name, m in build_critics(model_config).items():
if m is not None:
modules[f"{name}_critic"] = m
return modules
def _fingerprint(modules: dict[str, nn.Module]) -> list:
"""A config-shape fingerprint of the built graph: which submodules
exist, every parameter's/buffer's shape+dtype (never values those are
randomly initialized and irrelevant to *structure*), and every plain
scalar attribute any module stores on itself (e.g. `Stage2Autoregressive
.stop_sampling`, `EnergyRouter.temperature`) this is what makes a
non-parametric key's effect on construction observable."""
sig = []
for stage_name, module in modules.items():
for mod_name, m in module.named_modules():
full = f"{stage_name}.{mod_name}" if mod_name else stage_name
for k, v in vars(m).items():
if k.startswith("_"):
continue
if v is None or isinstance(v, (bool, int, float, str)):
sig.append((full, k, v))
for pname, p in module.named_parameters():
sig.append((stage_name, "param", pname, tuple(p.shape), str(p.dtype)))
for bname, b in module.named_buffers():
sig.append((stage_name, "buffer", bname, tuple(b.shape), str(b.dtype)))
return sorted(sig, key=repr)
def _perturb_candidates(path: str, value) -> list:
"""Values to try perturbing `path`'s current `value` to, in order —
probing stops at the first one that changes the fingerprint or raises.
Almost always a single candidate; see `_STRING_ALTERNATIVES`."""
if isinstance(value, bool):
return [not value]
if isinstance(value, int):
return [value + 1]
if isinstance(value, float):
return [value + 1.0]
if isinstance(value, str):
alternatives = [v for v in _STRING_ALTERNATIVES.get(path, ()) if v != value]
return [*alternatives, _PROBE_STR]
raise TypeError(f"gitea #46 probing: unsupported leaf value type {type(value)!r} ({value!r})")
def _vocab_caveats(cfg: dict) -> list[str]:
caveats = []
if _get_path(cfg, "conditioning.particle.type") == "embedding":
caveats.append(
"conditioning.particle.type = 'embedding' -- pdg_vocab below is a "
"placeholder (no dataset attached to derive the real training vocab size)"
)
if _get_path(cfg, "conditioning.material.type") == "embedding":
caveats.append(
"conditioning.material.type = 'embedding' -- mat_vocab below is a "
"placeholder (no dataset attached to derive the real training vocab size)"
)
for stage in ("stage1_model", "stage2_model"):
router_type = _get_path(cfg, f"{stage}.router.type")
if _get_path(cfg, f"{stage}.router.enabled") and router_type in ("pdg", "process"):
caveats.append(
f"{stage}.router.type = {router_type!r} builds its own pdg_vocab-sized "
"embedding -- the count above is a placeholder"
)
return caveats
def summarize_model(cfg: dict, pdg_vocab: int, mat_vocab: int) -> ModelSummary:
"""Build `cfg`'s model with no dataset attached and report its resolved
graph, plus which `conditioning`/`stage1_model`/`stage2_model` config
keys actually shaped it (differential probing see module docstring).
`cfg` must already be a fully-merged v0.3 config (`merge_cli_overrides`
output) this does not migrate or validate it."""
modules = _built_modules(cfg, pdg_vocab, mat_vocab)
baseline_fp = _fingerprint(modules)
in_scope = [p for p in leaf_paths(cfg) if p.split(".", 1)[0] in _IN_SCOPE_ROOTS]
consumed: list[str] = []
inert: list[str] = []
elsewhere: list[str] = []
for path in in_scope:
original = _get_path(cfg, path)
changed = False
for candidate in _perturb_candidates(path, original):
probe_cfg = copy.deepcopy(
{
"conditioning": cfg["conditioning"],
"stage1_model": cfg["stage1_model"],
"stage2_model": cfg["stage2_model"],
}
)
_set_path(probe_cfg, path, candidate)
try:
changed = _fingerprint(_built_modules(probe_cfg, pdg_vocab, mat_vocab)) != baseline_fp
except Exception:
changed = True
if changed:
break
if changed:
consumed.append(path)
elif path in _NOT_BUILD_TIME:
elsewhere.append(path)
else:
inert.append(path)
return ModelSummary(
modules=modules,
consumed=sorted(consumed),
inert=sorted(inert),
elsewhere=sorted(elsewhere),
pdg_vocab=pdg_vocab,
mat_vocab=mat_vocab,
vocab_caveats=_vocab_caveats(cfg),
)
def _tree_lines(module: nn.Module, name: str, indent: int = 0) -> list[str]:
total = sum(p.numel() for p in module.parameters())
in_dim = getattr(module, "in_dim", None)
out_dim = getattr(module, "out_dim", None)
widths = f" [in={in_dim}, out={out_dim}]" if in_dim is not None and out_dim is not None else ""
lines = [f"{' ' * indent}{name} ({type(module).__name__}): {total:,}{widths}"]
for child_name, child in module.named_children():
lines.extend(_tree_lines(child, child_name, indent + 1))
return lines
_HEAD_NAMES = ("n_sec_head", "type_head", "stop_head")
def _stage_header(name: str, module: nn.Module) -> list[str]:
total = sum(p.numel() for p in module.parameters())
lines = [f"{name}: {type(module).__name__} -- {total:,} parameters"]
generator = getattr(module, "generator_kind", None)
if generator is not None:
lines.append(f" generator: {generator}")
trunk = getattr(module, "trunk", None)
if trunk is not None:
in_dim = getattr(trunk, "in_dim", "?")
out_dim = getattr(trunk, "out_dim", "?")
if isinstance(trunk, RoutedTrunk):
detail = f"routed, n_experts={trunk.router.n_experts}, expert type={type(trunk.experts[0]).__name__}"
else:
detail = f"unrouted, {type(trunk).__name__}"
lines.append(f" trunk: {detail}, in={in_dim}, out={out_dim}")
history_kind = getattr(module, "history_kind", None)
if history_kind is not None:
lines.append(f" autoregressive history: {history_kind}")
present = [h for h in _HEAD_NAMES if getattr(module, h, None) is not None]
absent = [h for h in _HEAD_NAMES if hasattr(module, h) and getattr(module, h) is None]
if present or absent:
lines.append(f" heads present: {', '.join(present) if present else 'none'}")
if absent:
lines.append(f" heads absent: {', '.join(absent)}")
return lines
def render_summary(summary: ModelSummary) -> str:
lines: list[str] = []
for name, module in summary.modules.items():
lines.extend(_stage_header(name, module))
lines.extend(_tree_lines(module, name, indent=1))
lines.append("")
lines.append(
f"config keys read during construction: {len(summary.consumed)} / "
f"read elsewhere (trainer/sampler/rollout): {len(summary.elsewhere)} / "
f"inert under this config: {len(summary.inert)}"
)
if summary.elsewhere:
lines.append("read elsewhere, not by construction:")
for path in summary.elsewhere:
lines.append(f" {path} ({_NOT_BUILD_TIME[path]})")
lines.append("inert under this config (declared, parsed, but doing nothing here):")
if summary.inert:
for path in summary.inert:
lines.append(f" {path}")
else:
lines.append(" (none)")
if summary.vocab_caveats:
lines.append("")
lines.append("vocab placeholder caveats:")
for caveat in summary.vocab_caveats:
lines.append(f" {caveat}")
return "\n".join(lines)
+85 -7
View File
@@ -73,6 +73,7 @@ class ExpertTrunk(nn.Module):
block_conditioning: str = "add", block_conditioning: str = "add",
) -> None: ) -> None:
super().__init__() super().__init__()
self.in_dim = in_dim
self.out_dim = out_dim self.out_dim = out_dim
self.input_proj = nn.Linear(in_dim, hidden_dim) self.input_proj = nn.Linear(in_dim, hidden_dim)
self.blocks = nn.ModuleList( self.blocks = nn.ModuleList(
@@ -93,6 +94,49 @@ class ExpertTrunk(nn.Module):
return self.out_proj(x) return self.out_proj(x)
@register_trunk("linear")
class LinearTrunk(nn.Module):
"""`nn.Linear(in_dim + cond_dim, out_dim)` over `concat([x, cond])` —
the trivial trunk body: no hidden layer, no ResBlock stack, no
nonlinearity. Ablates whether trunk depth/nonlinearity is earning its
parameters, holding everything else (heads, ConditionEncoder,
generator, ...) fixed. Composes for free with `router.enabled = true`
(gitea #33): a RoutedTrunk of n_experts linear bodies is "mixture of
trivial linear experts". `hidden_dim`/`n_blocks`/`dropout`/
`block_conditioning` are accepted and ignored, matching
`build_expert_body`'s shared factory signature.
`x` the trunk's own input (e.g. the noised primary vector for flow
matching) does not already carry conditioning; that's fused in
per-body via `cond`. So this concatenates `x` and `cond` itself to
remain a valid, conditioning-dependent model.
"""
def __init__(
self,
in_dim: int,
out_dim: int,
hidden_dim: int,
n_blocks: int,
cond_dim: int,
dropout: float = 0.0,
block_conditioning: str = "add",
) -> None:
super().__init__()
self.in_dim = in_dim
self.out_dim = out_dim
self.linear = nn.Linear(in_dim + cond_dim, out_dim)
def forward(
self,
x: torch.Tensor,
cond: torch.Tensor,
cond_cont: torch.Tensor | None = None,
cond_cat: torch.Tensor | None = None,
) -> torch.Tensor:
return self.linear(torch.cat([x, cond], dim=-1))
def _route_forward( def _route_forward(
experts: nn.ModuleList, experts: nn.ModuleList,
router: Router, router: Router,
@@ -108,21 +152,50 @@ def _route_forward(
N-expert dense compute, fully differentiable (`weight` is N-expert dense compute, fully differentiable (`weight` is
`router.combine_weights`). Eval mode: grouped top-1 dispatch each row `router.combine_weights`). Eval mode: grouped top-1 dispatch each row
runs exactly one expert, the actual source of the per-call speedup. runs exactly one expert, the actual source of the per-call speedup.
The accumulator's dtype is deferred to the first expert call rather than
fixed at fp32: under autocast (`train.precision = "bf16"`, gitea #47) an
expert's `ResBlock` stack returns bf16, and an fp32-fixed accumulator
would silently upcast every mixture term (train mode) or downcast every
dispatched row via `index_put_` (eval mode) making a `RoutedTrunk`
return a different dtype than the unrouted `ExpertTrunk` it's a drop-in
replacement for, purely because `router.enabled` was set.
`router.combine_weights` is deliberately fp32 internally (it forces its
own autocast-disabled region see `Router.combine_weights`'s docstring),
so `weights` itself is always fp32 regardless of the ambient precision.
Left as-is, `weights[:, i:i+1] * expert(x, cond)` would type-promote the
whole mixture back to fp32 by ordinary PyTorch promotion rules the same
dtype-mismatch bug this function exists to avoid, just moved one line
over. `weights` is cast down to each expert's own output dtype right
before combining: the softmax stays numerically stable at fp32, but its
*result* (values in [0, 1], not precision-sensitive to represent) loses
nothing meaningful by then being used at bf16.
""" """
if training: if training:
weights = router.combine_weights(cond_cont, cond_cat) # (B, n_experts) weights = router.combine_weights(cond_cont, cond_cat) # (B, n_experts), fp32
out = torch.zeros(x.shape[0], experts[0].out_dim, device=x.device) out = None
for i, expert in enumerate(experts): for i, expert in enumerate(experts):
out = out + weights[:, i : i + 1] * expert(x, cond) expert_out = expert(x, cond)
term = weights[:, i : i + 1].to(expert_out.dtype) * expert_out
out = term if out is None else out + term
assert out is not None, "RoutedTrunk built with zero experts"
return out return out
idx = router.top1(cond_cont, cond_cat) # (B,) idx = router.top1(cond_cont, cond_cat) # (B,)
out_dim = experts[0].out_dim out = None
out = torch.zeros(x.shape[0], out_dim, device=x.device)
for i, expert in enumerate(experts): for i, expert in enumerate(experts):
mask = idx == i mask = idx == i
if mask.any(): if mask.any():
out[mask] = expert(x[mask], cond[mask]) expert_out = expert(x[mask], cond[mask])
if out is None:
out = torch.zeros(x.shape[0], expert_out.shape[-1], device=x.device, dtype=expert_out.dtype)
out[mask] = expert_out
if out is None:
# No row was ever dispatched (only reachable with an empty batch,
# x.shape[0] == 0) — nothing to infer a dtype from, so fall back to
# x's own, matching this function's pre-autocast behavior.
out = torch.zeros(x.shape[0], experts[0].out_dim, device=x.device, dtype=x.dtype)
return out return out
@@ -130,7 +203,10 @@ class Trunk(nn.Module):
"""Interface implemented by a standalone trunk body (any `TRUNK_REGISTRY` """Interface implemented by a standalone trunk body (any `TRUNK_REGISTRY`
entry, e.g. `ExpertTrunk`) and by `RoutedTrunk`: everything downstream of entry, e.g. `ExpertTrunk`) and by `RoutedTrunk`: everything downstream of
the fused conditioning vector, i.e. the actual generative trunk of a the fused conditioning vector, i.e. the actual generative trunk of a
stage.""" stage. Implementations are expected to expose `in_dim`/`out_dim`
attributes (as `ExpertTrunk`/`RoutedTrunk` do) `giant.model.summary`
(gitea #46) reads them to report trunk widths without needing to know the
body architecture."""
def forward( def forward(
self, self,
@@ -157,6 +233,8 @@ class RoutedTrunk(Trunk):
) -> None: ) -> None:
super().__init__() super().__init__()
self.router = router self.router = router
self.in_dim = in_dim
self.out_dim = out_dim
self.experts = nn.ModuleList( self.experts = nn.ModuleList(
[ [
build_expert_body( build_expert_body(
+24 -14
View File
@@ -22,21 +22,31 @@ def gradient_penalty(
norm to 1 `x_hat`/`grad` are forced to all-zero for such a row, which norm to 1 `x_hat`/`grad` are forced to all-zero for such a row, which
would otherwise contribute a constant `(||0|| - 1)^2 == 1` bias to the would otherwise contribute a constant `(||0|| - 1)^2 == 1` bias to the
mean regardless of critic behavior so they're excluded from the mean. mean regardless of critic behavior so they're excluded from the mean.
Deliberately kept fp32 (`torch.autocast(..., enabled=False)`) regardless
of the caller's ambient `train.precision` autocast region: this is a
`create_graph=True` double-backward, and `grad.norm(2, dim=1)` sums
squares over the critic's full input width (hundreds of dims for stage
2), which overflows bf16's range at gradient magnitudes well within
normal early-WGAN-GP territory. Disclosed cost: the critic forward
inside this function always runs fp32, even when the rest of the WGAN
stage's step is bf16 (gitea #47).
""" """
eps = torch.rand(real.size(0), 1, device=real.device) with torch.autocast(real.device.type, enabled=False):
x_hat = eps * real + (1 - eps) * fake eps = torch.rand(real.size(0), 1, device=real.device)
if mask is not None: x_hat = eps * real.float() + (1 - eps) * fake.float()
x_hat = x_hat * mask if mask is not None:
x_hat = x_hat.requires_grad_(True) x_hat = x_hat * mask
scores = critic_fn(x_hat) x_hat = x_hat.requires_grad_(True)
grad = torch.autograd.grad(outputs=scores.sum(), inputs=x_hat, create_graph=True)[0] scores = critic_fn(x_hat)
if mask is not None: grad = torch.autograd.grad(outputs=scores.sum(), inputs=x_hat, create_graph=True)[0]
grad = grad * mask if mask is not None:
penalty = (grad.norm(2, dim=1) - 1) ** 2 grad = grad * mask
if mask is not None: penalty = (grad.norm(2, dim=1) - 1) ** 2
valid = (mask.sum(dim=1) > 0).float() if mask is not None:
return (penalty * valid).sum() / valid.sum().clamp_min(1.0) valid = (mask.sum(dim=1) > 0).float()
return penalty.mean() return (penalty * valid).sum() / valid.sum().clamp_min(1.0)
return penalty.mean()
def critic_loss( def critic_loss(
+1 -1
View File
@@ -359,7 +359,7 @@ def run_train_job(
"section)" "section)"
) )
config.validate_config(cfg) config.validate_config(cfg, resume=resume is not None)
particle_conditioning = cfg["conditioning"]["particle"]["type"] particle_conditioning = cfg["conditioning"]["particle"]["type"]
material_conditioning = cfg["conditioning"]["material"]["type"] material_conditioning = cfg["conditioning"]["material"]["type"]
k_max = cfg["stage2_model"]["k_max"] k_max = cfg["stage2_model"]["k_max"]
+2 -1
View File
@@ -5,7 +5,7 @@ Split out of the former single-module `giant/train.py`. The public surface is
that tests and tooling construct directly. that tests and tooling construct directly.
""" """
from giant.training.checkpoint import build_checkpoint, load_checkpoint from giant.training.checkpoint import build_checkpoint, init_stages_from_checkpoints, load_checkpoint
from giant.training.metrics import MetricsCollector, MetricSpec from giant.training.metrics import MetricsCollector, MetricSpec
from giant.training.loop import train from giant.training.loop import train
from giant.training.trainers import ( from giant.training.trainers import (
@@ -25,6 +25,7 @@ __all__ = [
"WGANStageTrainer", "WGANStageTrainer",
"build_checkpoint", "build_checkpoint",
"build_stage_trainers", "build_stage_trainers",
"init_stages_from_checkpoints",
"load_checkpoint", "load_checkpoint",
"train", "train",
] ]
+47
View File
@@ -0,0 +1,47 @@
"""Mixed-precision training support (`train.precision`, gitea #47).
Only `"fp32"` (no autocast) and `"bf16"` are supported no `"fp16"`/
`GradScaler`. bf16 needs no gradient scaler and covers every training GPU in
the fleet (Ampere and newer: A100, L40S, H200, RTX 4070); fp16 would need a
scaler *and* fixes to two fragile spots that stay correct under bf16 but break
under fp16's narrower range — `giant.model.routers`' `1e-8` epsilons (below
fp16's ~6e-8 subnormal floor) and `giant.model.wgan.gradient_penalty`'s
sum-of-squares gradient norm (overflows fp16 above ~65504). Revisit if a
pre-Ampere (V100) training target ever shows up.
"""
import torch
_SUPPORTED_DEVICE_TYPES = ("cuda", "cpu")
def resolve_autocast(precision: str, device: torch.device) -> tuple[str, torch.dtype, bool]:
"""Resolves `train.precision` + a target device into the
`(device_type, dtype, enabled)` triple `torch.autocast` takes as kwargs
computed once per `StageTrainer` rather than re-derived every step.
Raises `ValueError` rather than silently falling back to fp32: a training
run that's quietly not using the mixed precision it was configured for is
a wasted GPU-week, not a warning.
"""
if precision == "fp32":
return device.type, torch.float32, False
if precision != "bf16":
raise ValueError(f"unknown precision {precision!r}; must be 'fp32' or 'bf16'")
if device.type == "cuda":
if not torch.cuda.is_bf16_supported():
cap = torch.cuda.get_device_capability(device)
raise ValueError(
f"train.precision = 'bf16' but {torch.cuda.get_device_name(device)} "
f"(compute capability {cap[0]}.{cap[1]}) has no native bf16 support "
"(needs Ampere/sm_80 or newer) — use train.precision = 'fp32' instead"
)
return "cuda", torch.bfloat16, True
if device.type == "cpu":
# torch 2.3's CPU autocast supports bf16 unconditionally — this is
# also what lets the bf16 training path be tested without a GPU.
return "cpu", torch.bfloat16, True
raise ValueError(
f"train.precision = 'bf16' is not supported on device type {device.type!r} (only {_SUPPORTED_DEVICE_TYPES} are)"
)
+33
View File
@@ -8,6 +8,8 @@ and per-stage `optimizer_<stage>` / `optimizer_d_<stage>` / `lr_sched_<stage>`
entries. entries.
""" """
import torch
from giant.training.trainers import StageTrainer from giant.training.trainers import StageTrainer
#: Stage name -> the checkpoint key its weights live under. Historical: stage #: Stage name -> the checkpoint key its weights live under. Historical: stage
@@ -46,6 +48,37 @@ def build_checkpoint(
return ckpt return ckpt
def init_stages_from_checkpoints(trainers: dict[str, StageTrainer]) -> list[str]:
"""Load each trainer's `spec.init_from` checkpoint (gitea #42) into its
model, before training starts the partial-retrain counterpart to
`load_checkpoint`'s full-run `--resume`. Only weights move: unlike
`load_checkpoint`, this never touches optimizer/lr_sched/epoch state, so
it composes cleanly with `--resume` (call this first; a resume's own
`load_checkpoint` then overwrites whatever this loaded with the resumed
run's own weights).
A stage with no `init_from` set (`""`, the default) is left alone. The
EMA companion (`<key>_ema`) is loaded too when both the source checkpoint
and this trainer have one, so `--weights ema` at inference still sees the
source's EMA shadow rather than a copy of its raw weights. Returns one
description string per stage actually initialized, for the caller to
echo.
"""
loaded = []
for name, trainer in trainers.items():
init_from = trainer.spec.init_from
if not init_from:
continue
key = _STAGE_KEY[name]
ckpt = torch.load(init_from, map_location="cpu", weights_only=False)
trainer.model.load_state_dict(ckpt[key])
ema_key = f"{key}_ema"
if trainer.ema_model is not None and ema_key in ckpt:
trainer.ema_model.load_state_dict(ckpt[ema_key])
loaded.append(f"{name}: loaded from {init_from}" + (" (frozen)" if trainer.frozen else ""))
return loaded
def load_checkpoint(trainers: dict[str, StageTrainer], ckpt: dict, lr: float) -> None: def load_checkpoint(trainers: dict[str, StageTrainer], ckpt: dict, lr: float) -> None:
"""Restore every active stage, then hand `lr`'s authority back to the """Restore every active stage, then hand `lr`'s authority back to the
config `load_state_dict` would otherwise leave the checkpoint's own config `load_state_dict` would otherwise leave the checkpoint's own
+5 -2
View File
@@ -20,7 +20,7 @@ from tqdm import tqdm
from giant.data.loader import TopNMap from giant.data.loader import TopNMap
from giant.data.setup_cache import topnmap_to_json from giant.data.setup_cache import topnmap_to_json
from giant.training.checkpoint import build_checkpoint, load_checkpoint from giant.training.checkpoint import build_checkpoint, init_stages_from_checkpoints, load_checkpoint
from giant.training.metrics import MetricsCollector from giant.training.metrics import MetricsCollector
from giant.training.trainers import ( from giant.training.trainers import (
FlowDDPMStageTrainer, FlowDDPMStageTrainer,
@@ -132,9 +132,12 @@ def train(
validate_steps = t.get("validate_steps", 10) validate_steps = t.get("validate_steps", 10)
max_val_batches = t.get("max_val_batches", 0) max_val_batches = t.get("max_val_batches", 0)
trainers = build_stage_trainers(cfg, models, critics, device, total_train_batches) sec_type_class_counts = sec_type_topn_map.class_counts if sec_type_topn_map is not None else None
trainers = build_stage_trainers(cfg, models, critics, device, total_train_batches, sec_type_class_counts)
if not trainers: if not trainers:
raise ValueError("no active stage — stage1_model.active and stage2_model.active are both false") raise ValueError("no active stage — stage1_model.active and stage2_model.active are both false")
for line in init_stages_from_checkpoints(trainers):
print(line)
has_adversarial = any(not tr.supports_val_loss for tr in trainers.values()) has_adversarial = any(not tr.supports_val_loss for tr in trainers.values())
checkpoint_extras = { checkpoint_extras = {
+33 -9
View File
@@ -120,9 +120,17 @@ def _remaining_energy_fraction(fraction: torch.Tensor) -> torch.Tensor:
slot i: `1.0` at `i=0`, `prod_{j<i}(1-fraction_j)` for `i>=1` slot i: `1.0` at `i=0`, `prod_{j<i}(1-fraction_j)` for `i>=1`
("no re-derivation needed": the existing ("no re-derivation needed": the existing
stick-breaking encoding is already scale-free, so this is derivable from stick-breaking encoding is already scale-free, so this is derivable from
the batch's ground-truth stick logits alone, no `e_sec` required).""" the batch's ground-truth stick logits alone, no `e_sec` required).
cumprod = torch.cumprod(1.0 - fraction, dim=1)
return torch.cat([torch.ones_like(cumprod[:, :1]), cumprod[:, :-1]], dim=1) Forced fp32 regardless of the caller's ambient `train.precision` autocast
region: a `cumprod` over `K_MAX` slots in bf16 underflows to zero within a
handful of slots, killing `remaining_frac` as a conditioning signal the
numpy encoder (`giant.data.transforms.encode_secondaries`'s stick-breaking
twin) already promotes to float64 for exactly this reason (gitea #47)."""
with torch.autocast(fraction.device.type, enabled=False):
fraction = fraction.float()
cumprod = torch.cumprod(1.0 - fraction, dim=1)
return torch.cat([torch.ones_like(cumprod[:, :1]), cumprod[:, :-1]], dim=1)
def _shift_prev(x: torch.Tensor) -> torch.Tensor: def _shift_prev(x: torch.Tensor) -> torch.Tensor:
@@ -202,21 +210,37 @@ def _assemble_stage2_ar_inputs(
return {"history_feat": history_feat, **_ar_meta(K, B, device, fraction)} return {"history_feat": history_feat, **_ar_meta(K, B, device, fraction)}
def _linear_schedule(p_start: float, p_end: float, epoch: int, total_epochs: int) -> float:
"""Linear interpolation from `p_start` (epoch 0) to `p_end` (the final
epoch) standard scheduled sampling (Bengio et al. 2015), shared by
every train-time schedule keyed on epoch."""
frac = epoch / max(total_epochs - 1, 1)
frac = min(max(frac, 0.0), 1.0)
return p_start + (p_end - p_start) * frac
def _stage2_tf_prob(mode: str, p_start: float, p_end: float, epoch: int, total_epochs: int) -> float: def _stage2_tf_prob(mode: str, p_start: float, p_end: float, epoch: int, total_epochs: int) -> float:
"""P(condition slot k+1 on the TRUE token k rather than the model's own """P(condition slot k+1 on the TRUE token k rather than the model's own
prediction), for the current epoch prediction), for the current epoch
(`stage2_model.autoregressive.teacher_forcing`). (`stage2_model.autoregressive.teacher_forcing`).
`"always"`/`"never"` are the two degenerate constants; `"scheduled"` `"always"`/`"never"` are the two degenerate constants; `"scheduled"`
linearly interpolates linearly interpolates `p_start` to `p_end` via `_linear_schedule`."""
`p_start` (epoch 0) to `p_end` (the final epoch) standard scheduled
sampling (Bengio et al. 2015)."""
if mode == "always": if mode == "always":
return 1.0 return 1.0
if mode == "never": if mode == "never":
return 0.0 return 0.0
frac = epoch / max(total_epochs - 1, 1) return _linear_schedule(p_start, p_end, epoch, total_epochs)
frac = min(max(frac, 0.0), 1.0)
return p_start + (p_end - p_start) * frac
def _ctx_truth_prob(mode: str, p_start: float, p_end: float, epoch: int, total_epochs: int) -> float:
"""P(condition stage 2 on the TRUE stage-1 outcome rather than a fresh
stage-1 sample), for the current epoch (`stage2_model.stage1_context`).
`"truth"` is the degenerate constant 1.0; `"sampled"` linearly
interpolates `ctx_p_start` to `ctx_p_end` via `_linear_schedule` the
stage-boundary counterpart of `_stage2_tf_prob`."""
if mode == "truth":
return 1.0
return _linear_schedule(p_start, p_end, epoch, total_epochs)
def _history_repr_from_ar_sample( def _history_repr_from_ar_sample(
+275 -68
View File
@@ -27,10 +27,13 @@ from giant.constants import CONT_SLOT_DIM
from giant.data.dataset import StepBatch from giant.data.dataset import StepBatch
from giant.model.network import Router, build_objective, resolve_type_n_classes, stage2_type_dim from giant.model.network import Router, build_objective, resolve_type_n_classes, stage2_type_dim
from giant.model.wgan import generator_loss, gradient_penalty from giant.model.wgan import generator_loss, gradient_penalty
from giant.sample import sample_stage1
from giant.training.amp import resolve_autocast
from giant.training.metrics import MetricSpec, stage_metric, train_metric, val_metric from giant.training.metrics import MetricSpec, stage_metric, train_metric, val_metric
from giant.training.stage2_inputs import ( from giant.training.stage2_inputs import (
_assemble_stage2_ar_inputs_scheduled, _assemble_stage2_ar_inputs_scheduled,
_assemble_stage2_ar_target, _assemble_stage2_ar_target,
_ctx_truth_prob,
_gumbel_tau, _gumbel_tau,
_relax_onehot_type_slice, _relax_onehot_type_slice,
_stage2_tf_prob, _stage2_tf_prob,
@@ -73,6 +76,41 @@ def _batch_to_device(batch: StepBatch, device: torch.device) -> StepBatch:
return type(batch)(*(t.to(device) for t in batch)) return type(batch)(*(t.to(device) for t in batch))
def _type_class_weight_vector(class_counts: dict[int, int], n_classes: int, scheme: str) -> list[float] | None:
"""Per-class `F.cross_entropy(weight=...)` vector for the stage-2 type
head's `class_weighting` (gitea #44), or `None` under `"none"` (the
pre-#44 unweighted-CE behavior — the caller must pass that through as
`weight=None`, not a vector of ones, so old runs stay bit-identical).
`"inverse_freq"`: `1 / count` per class, normalized to mean 1 over
`n_classes` so switching this on doesn't rescale the type loss against
`particle_type.lambda` / the generator loss it's summed with. A class
with zero training examples (fewer distinct species than `n_classes - 1`
slots) clamps its count to 1 its weight is otherwise undefined, and
since it never appears in a batch's labels the value is inert anyway.
Raises if `scheme != "none"` and `class_counts` is empty: that means the
`TopNMap` behind this run predates gitea #44 (a stale checkpoint's decode
map, or a not-yet-rebuilt setup-cache sidecar) and truly has no
frequency information to weight by silently falling back to uniform
weights would look like the feature is active when it isn't.
"""
if scheme == "none":
return None
if not class_counts:
raise ValueError(
f"stage2_model.particle_type.class_weighting = {scheme!r} requires "
"per-class counts, but this run's sec_type_topn_map has none "
"(class_counts={}) — it was built before gitea #44 or loaded "
"from a stale setup-cache sidecar/checkpoint; rebuild the setup "
"cache (giant train --rebuild-setup-cache) or retrain."
)
counts = [max(class_counts.get(i, 0), 1) for i in range(n_classes)]
inv = [1.0 / c for c in counts]
mean_inv = sum(inv) / len(inv)
return [w / mean_inv for w in inv]
@dataclass(frozen=True) @dataclass(frozen=True)
class StageSpec: class StageSpec:
"""One stage's resolved training configuration. """One stage's resolved training configuration.
@@ -87,6 +125,10 @@ class StageSpec:
generator: str generator: str
decoder: str = "one_shot" decoder: str = "one_shot"
# partial-retrain (gitea #42)
init_from: str = ""
freeze: bool = False
# loss weights # loss weights
lambda_weight: float = 1.0 lambda_weight: float = 1.0
n_sec_lambda: float = 0.1 n_sec_lambda: float = 0.1
@@ -95,6 +137,10 @@ class StageSpec:
# particle-type target (stage 2 only) # particle-type target (stage 2 only)
particle_type: ParticleTypeConfig = field(default_factory=ParticleTypeConfig) particle_type: ParticleTypeConfig = field(default_factory=ParticleTypeConfig)
particle_type_n_classes: int = 16 particle_type_n_classes: int = 16
# Resolved by from_config from sec_type_class_counts (dataset-derived,
# not itself a cfg value — see _type_class_weight_vector) crossed with
# particle_type.class_weighting (gitea #44). None under "none".
type_class_weights: list[float] | None = None
# optimization # optimization
lr: float = 3e-4 lr: float = 3e-4
@@ -103,6 +149,7 @@ class StageSpec:
warmup_epochs: int = 0 warmup_epochs: int = 0
epochs: int = 1 epochs: int = 1
steps_per_epoch: int = 1 steps_per_epoch: int = 1
precision: str = "fp32"
# routing auxiliaries # routing auxiliaries
lambda_balance: float = 0.0 lambda_balance: float = 0.0
@@ -117,6 +164,11 @@ class StageSpec:
tf_p_end: float = 1.0 tf_p_end: float = 1.0
ar_sample_steps: int = 10 ar_sample_steps: int = 10
# stage-1/stage-2 boundary (stage 2 only)
stage1_context: str = "truth"
ctx_p_start: float = 1.0
ctx_p_end: float = 0.0
# generator-specific # generator-specific
ddpm_n_steps: int = 1000 ddpm_n_steps: int = 1000
n_critic: int = 5 n_critic: int = 5
@@ -126,7 +178,18 @@ class StageSpec:
type_gumbel_tau_end: float = 0.1 type_gumbel_tau_end: float = 0.1
@classmethod @classmethod
def from_config(cls, cfg: dict, name: str, is_stage2: bool, steps_per_epoch: int) -> "StageSpec": def from_config(
cls,
cfg: dict,
name: str,
is_stage2: bool,
steps_per_epoch: int,
sec_type_class_counts: dict[int, int] | None = None,
) -> "StageSpec":
"""`sec_type_class_counts` is dataset-derived (`sec_type_topn_map.class_counts`,
gitea #44), not a `cfg` value — it's the one input to `StageSpec` that
doesn't come from `cfg`, kept separate from the "only place that
reads `cfg`" invariant below on purpose."""
t = TrainConfig.from_dict(cfg["train"]) t = TrainConfig.from_dict(cfg["train"])
# n_sec/particle_type/decoder/autoregressive/wgan's gumbel_tau_* are # n_sec/particle_type/decoder/autoregressive/wgan's gumbel_tau_* are
# stage-2-only concepts, always read off s2_spec (guarded by # stage-2-only concepts, always read off s2_spec (guarded by
@@ -139,17 +202,23 @@ class StageSpec:
# stage 1's). # stage 1's).
s2_spec = Stage2ModelConfig.from_dict(cfg["stage2_model"]) s2_spec = Stage2ModelConfig.from_dict(cfg["stage2_model"])
stage_spec = s2_spec if is_stage2 else Stage1ModelConfig.from_dict(cfg["stage1_model"]) stage_spec = s2_spec if is_stage2 else Stage1ModelConfig.from_dict(cfg["stage1_model"])
particle_type_n_classes = resolve_type_n_classes(
s2_spec.particle_type, cfg["conditioning"]["particle"]["emb_dim"]
)
return cls( return cls(
name=name, name=name,
is_stage2=is_stage2, is_stage2=is_stage2,
generator=stage_spec.generator, generator=stage_spec.generator,
decoder=s2_spec.decoder if is_stage2 else "one_shot", decoder=s2_spec.decoder if is_stage2 else "one_shot",
init_from=stage_spec.init_from,
freeze=stage_spec.freeze,
lambda_weight=stage_spec.lambda_weight, lambda_weight=stage_spec.lambda_weight,
n_sec_lambda=s2_spec.n_sec.lambda_weight, n_sec_lambda=s2_spec.n_sec.lambda_weight,
n_sec_mode=s2_spec.n_sec.mode, n_sec_mode=s2_spec.n_sec.mode,
particle_type=s2_spec.particle_type, particle_type=s2_spec.particle_type,
particle_type_n_classes=resolve_type_n_classes( particle_type_n_classes=particle_type_n_classes,
s2_spec.particle_type, cfg["conditioning"]["particle"]["emb_dim"] type_class_weights=_type_class_weight_vector(
sec_type_class_counts or {}, particle_type_n_classes, s2_spec.particle_type.class_weighting
), ),
# train.* keys are all guaranteed by DEFAULT_CONFIG's deep-merge # train.* keys are all guaranteed by DEFAULT_CONFIG's deep-merge
# (giant/config.py), so TrainConfig.from_dict never has to fall # (giant/config.py), so TrainConfig.from_dict never has to fall
@@ -161,6 +230,7 @@ class StageSpec:
warmup_epochs=t.warmup_epochs, warmup_epochs=t.warmup_epochs,
epochs=t.epochs, epochs=t.epochs,
steps_per_epoch=max(steps_per_epoch, 1), steps_per_epoch=max(steps_per_epoch, 1),
precision=t.precision,
lambda_balance=stage_spec.router.lambda_balance, lambda_balance=stage_spec.router.lambda_balance,
lambda_proc=stage_spec.router.lambda_proc, lambda_proc=stage_spec.router.lambda_proc,
lambda_entropy=stage_spec.router.lambda_entropy, lambda_entropy=stage_spec.router.lambda_entropy,
@@ -169,6 +239,9 @@ class StageSpec:
teacher_forcing=s2_spec.autoregressive.teacher_forcing if is_stage2 else cls.teacher_forcing, teacher_forcing=s2_spec.autoregressive.teacher_forcing if is_stage2 else cls.teacher_forcing,
tf_p_start=s2_spec.autoregressive.tf_p_start if is_stage2 else cls.tf_p_start, tf_p_start=s2_spec.autoregressive.tf_p_start if is_stage2 else cls.tf_p_start,
tf_p_end=s2_spec.autoregressive.tf_p_end if is_stage2 else cls.tf_p_end, tf_p_end=s2_spec.autoregressive.tf_p_end if is_stage2 else cls.tf_p_end,
stage1_context=s2_spec.stage1_context if is_stage2 else cls.stage1_context,
ctx_p_start=s2_spec.ctx_p_start if is_stage2 else cls.ctx_p_start,
ctx_p_end=s2_spec.ctx_p_end if is_stage2 else cls.ctx_p_end,
# AR self-sampling under scheduled/never teacher forcing reuses # AR self-sampling under scheduled/never teacher forcing reuses
# train.validate_steps as its flow-matching ODE step count — no # train.validate_steps as its flow-matching ODE step count — no
# dedicated config key for this (the autoregressive config lists # dedicated config key for this (the autoregressive config lists
@@ -186,13 +259,16 @@ class StageSpec:
class StageTrainer: class StageTrainer:
"""One active stage's optimizer(s), EMA, and per-batch step. """One active stage's optimizer(s), EMA, and per-batch step.
Reads only the shared `StepBatch` (`giant.data.dataset`) stage 2 always Reads only the shared `StepBatch` (`giant.data.dataset`) by default stage
conditions on the ground-truth `x1_s1` (`stage2_model.stage1_context = 2 conditions on the ground-truth `x1_s1` (`stage2_model.stage1_context =
"truth"`, stage-level teacher forcing; `"sampled"` is not implemented), "truth"`, stage-level teacher forcing), so "stage-2-only training is a
so stage trainers never need each other's output at train time. This means cheap ablation, not new plumbing" falls out for free: a trainer only
"stage-2-only training is a cheap ablation, not new plumbing" falls out exists for active stages, and inactive stages are simply never
for free: a trainer only exists for active stages, and inactive stages constructed. `stage2_model.stage1_context = "sampled"` is the one
are simply never constructed. exception `build_stage_trainers` wires the stage-2 trainer to the
stage-1 one via `attach_stage1` so it can draw a real stage-1 sample
(`giant.sample.sample_stage1`) instead, scheduled by `ctx_p_start`/
`ctx_p_end` (see `_stage1_context`).
Grad-norm clipping is per-stage here v0.2's single shared optimizer Grad-norm clipping is per-stage here v0.2's single shared optimizer
clipped both stages' gradients jointly; splitting per stage is a small, clipped both stages' gradients jointly; splitting per stage is a small,
@@ -231,11 +307,18 @@ class StageTrainer:
self.is_stage2 = spec.is_stage2 self.is_stage2 = spec.is_stage2
self.generator = spec.generator self.generator = spec.generator
self.decoder = spec.decoder self.decoder = spec.decoder
self.frozen = spec.freeze
self.device = device self.device = device
self.model = model.to(device) self.model = model.to(device)
self.router = _stage_router(self.model) self.router = _stage_router(self.model)
self._modules = (self.model, *extra_modules) self._modules = (self.model, *extra_modules)
# Resolved once (not re-derived every step) — see
# giant.training.amp.resolve_autocast (gitea #47).
self._autocast_device_type, self._autocast_dtype, self._autocast_enabled = resolve_autocast(
spec.precision, device
)
self.particle_type_cfg = spec.particle_type self.particle_type_cfg = spec.particle_type
self.particle_type_n_classes = spec.particle_type_n_classes self.particle_type_n_classes = spec.particle_type_n_classes
self.ema_decay = spec.ema_decay self.ema_decay = spec.ema_decay
@@ -246,6 +329,18 @@ class StageTrainer:
for p in self.ema_model.parameters(): for p in self.ema_model.parameters():
p.requires_grad_(False) p.requires_grad_(False)
#: Set by `attach_stage1` when `stage2_model.stage1_context =
#: "sampled"` — the stage-1 `StageTrainer` this (stage-2) trainer
#: draws its context sample from. `None` for stage 1 itself, and for
#: stage 2 under "truth".
self.stage1_source: "StageTrainer | None" = None
def attach_stage1(self, stage1_trainer: "StageTrainer") -> None:
"""Wires this (stage-2) trainer to the stage-1 trainer it should
sample from under `stage2_model.stage1_context = "sampled"` see
`build_stage_trainers`."""
self.stage1_source = stage1_trainer
# --- schedule ------------------------------------------------------- # --- schedule -------------------------------------------------------
def _init_lr_schedule(self, optimizer: optim.Optimizer, warmup_steps: int, total_steps: int) -> None: def _init_lr_schedule(self, optimizer: optim.Optimizer, warmup_steps: int, total_steps: int) -> None:
@@ -289,6 +384,58 @@ class StageTrainer:
for module in self._modules: for module in self._modules:
module.eval() module.eval()
# --- stage-1/stage-2 boundary (shared by both trainer subclasses) ---
def _stage1_context(
self,
x1_s1: torch.Tensor,
cond_cont: torch.Tensor,
cond_cat: torch.Tensor,
epoch: int | None,
) -> torch.Tensor:
"""The stage-1 outcome stage 2 conditions on this batch.
`epoch=None` means "always ground truth" regardless of
`spec.stage1_context` the same val-loss convention `_ar_inputs`
uses, so validation stays a stable, non-stochastic comparison.
Otherwise, under `stage1_context = "sampled"`, each example
independently uses the ground truth with probability `p_truth`
(`_ctx_truth_prob`, ramped by `ctx_p_start`/`ctx_p_end`) and a fresh
`giant.sample.sample_stage1` draw from `stage1_source.sampling_model()`
otherwise a real sampling pass, not a cheap proxy, matching
`_assemble_stage2_ar_inputs_scheduled`'s precedent for the equivalent
in-stage-2 self-sample. Mixed per example (not per-dimension): a row
is either the real ground-truth 9D vector or a real sample, never an
elementwise blend of the two.
"""
x1_s1 = x1_s1.detach()
if self.stage1_source is None or epoch is None:
return x1_s1
p_truth = _ctx_truth_prob(
self.spec.stage1_context,
self.spec.ctx_p_start,
self.spec.ctx_p_end,
epoch,
self.spec.epochs,
)
if p_truth >= 1.0:
return x1_s1
stage1_model = self.stage1_source.sampling_model()
was_training = stage1_model.training
sampled, _ = sample_stage1(
stage1_model,
cond_cont,
cond_cat,
steps=self.spec.ar_sample_steps,
ddpm_steps=self.stage1_source.spec.ddpm_n_steps,
)
if was_training:
stage1_model.train()
use_truth = torch.rand(x1_s1.size(0), 1, device=x1_s1.device) < p_truth
return torch.where(use_truth, x1_s1, sampled).detach()
# --- stage-2 secondary assembly (shared by both trainer subclasses) --- # --- stage-2 secondary assembly (shared by both trainer subclasses) ---
def _ar_inputs( def _ar_inputs(
@@ -434,14 +581,36 @@ class StageTrainer:
stop_acc = (((logits >= 0).float() == target).float() * mask_f).sum() / denom stop_acc = (((logits >= 0).float() == target).float() * mask_f).sum() / denom
return l_stop, stop_acc return l_stop, stop_acc
@staticmethod def _autocast(self) -> torch.autocast:
def _step_optimizer(optimizer: optim.Optimizer, loss: torch.Tensor, params: list) -> float: """The training-step autocast region (`train.precision`, gitea #47).
Only wraps forward/loss computation `backward()`/`optimizer.step()`
stay outside, and `val_loss` never calls this at all, so validation
(and the best-checkpoint selection it drives) stays precision-
independent and comparable against every fp32-only run recorded so
far. `enabled=False` under `precision = "fp32"` (the default) makes
this a true no-op, so callers never need to branch on precision
themselves."""
return torch.autocast(
self._autocast_device_type,
dtype=self._autocast_dtype,
enabled=self._autocast_enabled,
)
def _step_optimizer(self, optimizer: optim.Optimizer, loss: torch.Tensor, params: list) -> float:
"""`zero_grad -> backward -> clip_grad_norm_(1.0) -> step`, returning """`zero_grad -> backward -> clip_grad_norm_(1.0) -> step`, returning
the pre-clip grad norm. The one place the grad-clip constant lives.""" the pre-clip grad norm. The one place the grad-clip constant lives.
`self.frozen` (`stage{1,2}_model.freeze`, gitea #42) skips only the
final `optimizer.step()` backward/clip still run so loss/grad_norm
stay meaningful to watch, but the stage's weights (and, for a WGAN
stage, its critic's — this same method is both trainers' single
optimizer-step choke point) never move."""
optimizer.zero_grad() optimizer.zero_grad()
loss.backward() loss.backward()
grad_norm = torch.nn.utils.clip_grad_norm_(params, 1.0) grad_norm = torch.nn.utils.clip_grad_norm_(params, 1.0)
optimizer.step() if not self.frozen:
optimizer.step()
return grad_norm.item() return grad_norm.item()
def _extra_state(self) -> dict: def _extra_state(self) -> dict:
@@ -504,6 +673,12 @@ class FlowDDPMStageTrainer(StageTrainer):
# "onehot"/"embedding" pull it out into model.type_head instead (0 # "onehot"/"embedding" pull it out into model.type_head instead (0
# here). # here).
self._flow_type_dim = None if self.particle_type_cfg.target == "physical" else 0 self._flow_type_dim = None if self.particle_type_cfg.target == "physical" else 0
# gitea #44: None under class_weighting = "none" (the default),
# matching F.cross_entropy's own unweighted default — a real tensor
# only materializes when the config asked for one.
self.type_class_weights = (
None if spec.type_class_weights is None else torch.tensor(spec.type_class_weights, device=device)
)
self.params = list(self.model.parameters()) self.params = list(self.model.parameters())
self.optimizer = optim.AdamW(self.params, lr=spec.lr, weight_decay=spec.weight_decay) self.optimizer = optim.AdamW(self.params, lr=spec.lr, weight_decay=spec.weight_decay)
@@ -597,8 +772,12 @@ class FlowDDPMStageTrainer(StageTrainer):
mask = sec_mask.float() mask = sec_mask.float()
denom = mask.sum().clamp(min=1) denom = mask.sum().clamp(min=1)
if self.particle_type_cfg.target == "onehot": if self.particle_type_cfg.target == "onehot":
ce = F.cross_entropy(type_out.transpose(1, 2), sec_type_idx, reduction="none") ce = F.cross_entropy(
type_out.transpose(1, 2), sec_type_idx, weight=self.type_class_weights, reduction="none"
)
l_type = (ce * mask).sum() / denom l_type = (ce * mask).sum() / denom
# Unweighted, deliberately — type_acc is a diagnostic of raw
# per-slot correctness, not the (possibly class-weighted) loss.
type_acc = ((type_out.argmax(-1) == sec_type_idx).float() * mask).sum() / denom type_acc = ((type_out.argmax(-1) == sec_type_idx).float() * mask).sum() / denom
else: # "embedding" else: # "embedding"
target_vec = self.model.cond_enc.pdg_emb(sec_type_idx).detach() target_vec = self.model.cond_enc.pdg_emb(sec_type_idx).detach()
@@ -608,9 +787,10 @@ class FlowDDPMStageTrainer(StageTrainer):
def _compute(self, batch: StepBatch, device: torch.device, epoch: int | None = None) -> dict: def _compute(self, batch: StepBatch, device: torch.device, epoch: int | None = None) -> dict:
"""`epoch=None` (the `val_loss` path) always uses full teacher """`epoch=None` (the `val_loss` path) always uses full teacher
forcing (`p_tf=1.0`) regardless of `spec.teacher_forcing` validation forcing (`p_tf=1.0`) and the ground-truth stage-1 context, regardless
should stay a stable, non-stochastic ground-truth comparison; only of `spec.teacher_forcing`/`spec.stage1_context` validation should
the training `step` path schedules `p_tf` by epoch.""" stay a stable, non-stochastic ground-truth comparison; only the
training `step` path schedules `p_tf`/`p_truth` by epoch."""
( (
cond_cont, cond_cont,
cond_cat, cond_cat,
@@ -621,7 +801,7 @@ class FlowDDPMStageTrainer(StageTrainer):
sec_type_idx, sec_type_idx,
) = _batch_to_device(batch, device) ) = _batch_to_device(batch, device)
sec_mask = self._sec_mask(n_sec, sec_cont.size(1), device) sec_mask = self._sec_mask(n_sec, sec_cont.size(1), device)
stage1_ctx = x1_s1.detach() stage1_ctx = self._stage1_context(x1_s1, cond_cont, cond_cat, epoch)
x1_s2 = None x1_s2 = None
ar_inputs = None ar_inputs = None
@@ -689,10 +869,12 @@ class FlowDDPMStageTrainer(StageTrainer):
self.spec.gumbel_tau_end, self.spec.gumbel_tau_end,
) )
epoch = global_step // self.spec.steps_per_epoch epoch = global_step // self.spec.steps_per_epoch
out = self._compute(batch, device, epoch=epoch) with self._autocast():
out = self._compute(batch, device, epoch=epoch)
grad_norm = self._step_optimizer(self.optimizer, out["loss"], self.params) grad_norm = self._step_optimizer(self.optimizer, out["loss"], self.params)
self.lr_sched.step() if not self.frozen:
if self.ema_model is not None: self.lr_sched.step()
if self.ema_model is not None and not self.frozen:
_update_ema(self.ema_model, self.model, self.ema_decay) _update_ema(self.ema_model, self.model, self.ema_decay)
stats = {key: value.item() for key, value in out.items()} stats = {key: value.item() for key, value in out.items()}
stats["grad_norm"] = grad_norm stats["grad_norm"] = grad_norm
@@ -846,53 +1028,58 @@ class WGANStageTrainer(StageTrainer):
sec_type_idx, sec_type_idx,
) = _batch_to_device(batch, device) ) = _batch_to_device(batch, device)
B = cond_cont.size(0) B = cond_cont.size(0)
stage1_ctx = x1_s1.detach() epoch = global_step // self.spec.steps_per_epoch
stage1_ctx = self._stage1_context(x1_s1, cond_cont, cond_cat, epoch)
grad_probe: dict[str, float] = {} grad_probe: dict[str, float] = {}
ar_inputs = None ar_inputs = None
if not self.is_stage2: with self._autocast():
real = x1_s1 if not self.is_stage2:
real = x1_s1
def critic_fn(x): def critic_fn(x):
return self.critic(x, cond_cont, cond_cat) return self.critic(x, cond_cont, cond_cat)
z = torch.randn(B, self.model.noise_dim, device=device) z = torch.randn(B, self.model.noise_dim, device=device)
fake = self.model(z, cond_cont, cond_cat) fake = self.model(z, cond_cont, cond_cat)
mask = None mask = None
else: else:
real, fake_raw, mask, critic_fn, ar_inputs = self._stage2_real_and_fake( real, fake_raw, mask, critic_fn, ar_inputs = self._stage2_real_and_fake(
_Stage2RealFakeBatch(cond_cont, cond_cat, n_sec, sec_cont, sec_type_idx), _Stage2RealFakeBatch(cond_cont, cond_cat, n_sec, sec_cont, sec_type_idx),
stage1_ctx, stage1_ctx,
global_step,
device,
)
if self.particle_type_cfg.target == "onehot":
# Straight-through Gumbel-softmax relaxation of the type
# slice only — the critic must see a hard one-hot forward
# (matching what "real" data looks like) while gradient
# still flows smoothly to the generator. grad_probe captures
# the gradient-magnitude instrumentation — see
# _relax_onehot_type_slice's docstring.
tau = _gumbel_tau(
global_step, global_step,
self.total_steps, device,
self.spec.type_gumbel_tau_start,
self.spec.type_gumbel_tau_end,
) )
fake_raw = _relax_onehot_type_slice( if self.particle_type_cfg.target == "onehot":
fake_raw, # Straight-through Gumbel-softmax relaxation of the type
sec_cont.size(1), # slice only — the critic must see a hard one-hot forward
CONT_SLOT_DIM, # (matching what "real" data looks like) while gradient
stage2_type_dim(self.particle_type_cfg, self.particle_type_n_classes), # still flows smoothly to the generator. grad_probe captures
tau, # the gradient-magnitude instrumentation — see
grad_probe=grad_probe, # _relax_onehot_type_slice's docstring.
) tau = _gumbel_tau(
fake = fake_raw * mask global_step,
self.total_steps,
self.spec.type_gumbel_tau_start,
self.spec.type_gumbel_tau_end,
)
fake_raw = _relax_onehot_type_slice(
fake_raw,
sec_cont.size(1),
CONT_SLOT_DIM,
stage2_type_dim(self.particle_type_cfg, self.particle_type_n_classes),
tau,
grad_probe=grad_probe,
)
fake = fake_raw * mask
# --- critic step (every batch) --- # --- critic step (every batch) ---
fake_detached = fake.detach() fake_detached = fake.detach()
real_score = critic_fn(real) real_score = critic_fn(real)
fake_score = critic_fn(fake_detached) fake_score = critic_fn(fake_detached)
# gradient_penalty forces its own fp32 region internally (see its
# docstring) regardless of the ambient autocast above.
gp = gradient_penalty(critic_fn, real, fake_detached, mask=mask) gp = gradient_penalty(critic_fn, real, fake_detached, mask=mask)
d_loss = fake_score.mean() - real_score.mean() + self.gp_weight * gp d_loss = fake_score.mean() - real_score.mean() + self.gp_weight * gp
wasserstein = (real_score.mean() - fake_score.mean()).detach() wasserstein = (real_score.mean() - fake_score.mean()).detach()
@@ -901,8 +1088,9 @@ class WGANStageTrainer(StageTrainer):
# --- generator (+ n_sec) step --- # --- generator (+ n_sec) step ---
did_g_step = global_step % self.n_critic == 0 did_g_step = global_step % self.n_critic == 0
l_nsec, nsec_acc = self._n_sec_loss(cond_cont, cond_cat, stage1_ctx, n_sec, device) with self._autocast():
l_stop, stop_acc = self._stop_loss(cond_cont, cond_cat, stage1_ctx, n_sec, device, ar_inputs) l_nsec, nsec_acc = self._n_sec_loss(cond_cont, cond_cat, stage1_ctx, n_sec, device)
l_stop, stop_acc = self._stop_loss(cond_cont, cond_cat, stage1_ctx, n_sec, device, ar_inputs)
# On a non-generator-step batch with no n_sec_head/stop_head on this # On a non-generator-step batch with no n_sec_head/stop_head on this
# stage (n_sec now defaults to stage 2), there's nothing for # stage (n_sec now defaults to stage 2), there's nothing for
@@ -910,7 +1098,8 @@ class WGANStageTrainer(StageTrainer):
# be a graph-less zero tensor, which .backward() rejects outright. # be a graph-less zero tensor, which .backward() rejects outright.
skip_g_step = not did_g_step and self.model.n_sec_head is None and self.model.stop_head is None skip_g_step = not did_g_step and self.model.n_sec_head is None and self.model.stop_head is None
if did_g_step: if did_g_step:
g_loss_adv = generator_loss(critic_fn, fake) with self._autocast():
g_loss_adv = generator_loss(critic_fn, fake)
g_loss = self.spec.lambda_weight * g_loss_adv + self.spec.n_sec_lambda * (l_nsec + l_stop) g_loss = self.spec.lambda_weight * g_loss_adv + self.spec.n_sec_lambda * (l_nsec + l_stop)
else: else:
g_loss_adv = torch.zeros((), device=device) g_loss_adv = torch.zeros((), device=device)
@@ -921,8 +1110,9 @@ class WGANStageTrainer(StageTrainer):
grad_norm_g = self._step_optimizer(self.optimizer, g_loss, self.g_params) grad_norm_g = self._step_optimizer(self.optimizer, g_loss, self.g_params)
if did_g_step: if did_g_step:
self.lr_sched.step() if not self.frozen:
if self.ema_model is not None: self.lr_sched.step()
if self.ema_model is not None and not self.frozen:
_update_ema(self.ema_model, self.model, self.ema_decay) _update_ema(self.ema_model, self.model, self.ema_decay)
return { return {
@@ -994,15 +1184,27 @@ def build_stage_trainers(
critics: dict[str, torch.nn.Module | None], critics: dict[str, torch.nn.Module | None],
device: torch.device, device: torch.device,
total_train_batches: int, total_train_batches: int,
sec_type_class_counts: dict[int, int] | None = None,
) -> dict[str, StageTrainer]: ) -> dict[str, StageTrainer]:
"""One trainer per active stage — `models[name] is None` means that stage """One trainer per active stage — `models[name] is None` means that stage
is `active = false` and is simply never constructed.""" is `active = false` and is simply never constructed.
`stage2_model.stage1_context = "sampled"` additionally wires the
stage-2 trainer to the stage-1 one (`StageTrainer.attach_stage1`) so it
can draw a real stage-1 sample instead of only ever seeing the
ground-truth stage-1 outcome `validate_config` already guarantees both
stages are active whenever that config value is set.
`sec_type_class_counts` (`sec_type_topn_map.class_counts`, gitea #44) is
the one dataset-derived input `StageSpec.from_config` needs beyond `cfg`
`None`/absent whenever `stage2_model.particle_type.class_weighting =
"none"` (the default), which never reads it."""
trainers: dict[str, StageTrainer] = {} trainers: dict[str, StageTrainer] = {}
for name, is_stage2 in (("stage1", False), ("stage2", True)): for name, is_stage2 in (("stage1", False), ("stage2", True)):
model = models.get(name) model = models.get(name)
if model is None: if model is None:
continue continue
spec = StageSpec.from_config(cfg, name, is_stage2, max(total_train_batches, 1)) spec = StageSpec.from_config(cfg, name, is_stage2, max(total_train_batches, 1), sec_type_class_counts)
if build_objective(spec.generator).is_adversarial: if build_objective(spec.generator).is_adversarial:
critic = critics.get(name) critic = critics.get(name)
assert critic is not None, ( assert critic is not None, (
@@ -1011,4 +1213,9 @@ def build_stage_trainers(
trainers[name] = WGANStageTrainer(spec, model, critic, device) trainers[name] = WGANStageTrainer(spec, model, critic, device)
else: else:
trainers[name] = FlowDDPMStageTrainer(spec, model, device) trainers[name] = FlowDDPMStageTrainer(spec, model, device)
stage2 = trainers.get("stage2")
stage1 = trainers.get("stage1")
if stage2 is not None and stage1 is not None and stage2.spec.stage1_context == "sampled":
stage2.attach_stage1(stage1)
return trainers return trainers
+3 -1
View File
@@ -1,6 +1,6 @@
[project] [project]
name = "giant" name = "giant"
version = "0.3.2" version = "0.3.5"
description = "Geant4 step-function surrogate via conditional flow matching" description = "Geant4 step-function surrogate via conditional flow matching"
readme = "README.md" readme = "README.md"
requires-python = ">=3.12" requires-python = ">=3.12"
@@ -26,6 +26,8 @@ dev = [
"pytest-cov>=5,<8", "pytest-cov>=5,<8",
"ruff>=0.15,<1", "ruff>=0.15,<1",
"ty>=0.0.50,<0.1", "ty>=0.0.50,<0.1",
"bump-my-version>=1.2,<2",
"git-cliff>=2,<3",
"giant[convert,analysis,geometry,wandb]", "giant[convert,analysis,geometry,wandb]",
] ]
geometry = [ geometry = [
+136
View File
@@ -0,0 +1,136 @@
"""Tests for giant/training/amp.py (gitea #47)."""
import tempfile
from pathlib import Path
import pytest
import torch
from giant.model.routers import EnergyRouter
from giant.model.wgan import gradient_penalty
from giant.training.amp import resolve_autocast
from giant.training.stage2_inputs import _remaining_energy_fraction
from test_train import _base_cfg, _run_train
# ---------------------------------------------------------------------------
# resolve_autocast
# ---------------------------------------------------------------------------
def test_resolve_autocast_fp32_is_disabled():
device_type, dtype, enabled = resolve_autocast("fp32", torch.device("cpu"))
assert device_type == "cpu"
assert dtype is torch.float32
assert enabled is False
def test_resolve_autocast_bf16_on_cpu_is_enabled():
"""CPU bf16 autocast is what lets the mixed-precision path be tested
without a GPU (torch 2.3 supports it)."""
device_type, dtype, enabled = resolve_autocast("bf16", torch.device("cpu"))
assert device_type == "cpu"
assert dtype is torch.bfloat16
assert enabled is True
def test_resolve_autocast_bf16_on_unsupported_cuda_raises(monkeypatch):
monkeypatch.setattr(torch.cuda, "is_bf16_supported", lambda: False)
monkeypatch.setattr(torch.cuda, "get_device_capability", lambda device=None: (7, 0))
monkeypatch.setattr(torch.cuda, "get_device_name", lambda device=None: "Tesla V100")
with pytest.raises(ValueError, match="bf16"):
resolve_autocast("bf16", torch.device("cuda"))
def test_resolve_autocast_bf16_on_mps_raises():
with pytest.raises(ValueError, match="bf16"):
resolve_autocast("bf16", torch.device("mps"))
def test_resolve_autocast_unknown_precision_raises():
with pytest.raises(ValueError, match="fp32.*bf16"):
resolve_autocast("fp16", torch.device("cpu"))
# ---------------------------------------------------------------------------
# End-to-end: train() under bf16 on CPU
# ---------------------------------------------------------------------------
def test_train_end_to_end_bf16_cpu_completes_and_stores_fp32_params():
"""Reuses tests/test_train.py's synthetic-batch harness — train() itself
is device-agnostic, and CPU bf16 autocast is real (not mocked) in torch
2.3, so this is a genuine exercise of the autocast region added to
FlowDDPMStageTrainer.step/WGANStageTrainer.step, not just a config
passthrough check.
Also asserts the checkpoint's stored parameters are fp32: autocast only
changes the dtype of intermediate activations, never the model's own
stored weights a regression here would mean something accidentally
cast the model itself (e.g. `model.to(dtype=torch.bfloat16)`) rather than
using autocast."""
cfg = _base_cfg()
cfg["train"]["precision"] = "bf16"
with tempfile.TemporaryDirectory() as tmp:
out_dir = Path(tmp) / "run"
_run_train(cfg, out_dir)
assert (out_dir / "last.pt").exists()
assert (out_dir / "metrics.csv").exists()
ckpt = torch.load(out_dir / "last.pt", weights_only=False)
for stage_key in ("model", "sec_decoder"):
if stage_key not in ckpt:
continue
for name, tensor in ckpt[stage_key].items():
if tensor.is_floating_point():
assert tensor.dtype == torch.float32, f"{stage_key}.{name} is {tensor.dtype}, expected fp32"
@pytest.mark.parametrize("generator", ["wgan", "flow"])
def test_train_end_to_end_bf16_cpu_stage2_generators(generator):
"""bf16 covers both trainer subclasses (FlowDDPMStageTrainer and
WGANStageTrainer) the wgan default in _base_cfg exercises the
generator-forward/critic-scoring autocast region added to
WGANStageTrainer.step, and flow exercises the plain _compute wrap."""
cfg = _base_cfg()
cfg["train"]["precision"] = "bf16"
cfg["stage2_model"]["generator"] = generator
with tempfile.TemporaryDirectory() as tmp:
_run_train(cfg, Path(tmp) / "run")
# ---------------------------------------------------------------------------
# fp32 guards: correct in fp32, quietly degrade in bf16 — stay fp32 even
# under an active bf16 autocast region.
# ---------------------------------------------------------------------------
def test_remaining_energy_fraction_stays_fp32_under_bf16_autocast():
fraction = torch.rand(4, 5).to(torch.bfloat16)
with torch.autocast("cpu", dtype=torch.bfloat16, enabled=True):
out = _remaining_energy_fraction(fraction)
assert out.dtype == torch.float32
def test_gradient_penalty_stays_fp32_under_bf16_autocast():
critic = torch.nn.Linear(6, 1)
def critic_fn(x):
return critic(x)
real = torch.randn(4, 6)
fake = torch.randn(4, 6)
with torch.autocast("cpu", dtype=torch.bfloat16, enabled=True):
gp = gradient_penalty(critic_fn, real, fake)
assert gp.dtype == torch.float32
def test_router_balance_and_entropy_loss_stay_fp32_under_bf16_autocast():
router = EnergyRouter(n_experts=3)
cond_cont = torch.randn(8, 15)
cond_cat = torch.zeros(8, 2, dtype=torch.long)
with torch.autocast("cpu", dtype=torch.bfloat16, enabled=True):
balance = router.balance_loss(cond_cont, cond_cat)
entropy = router.entropy_loss(cond_cont, cond_cat)
weights = router.combine_weights(cond_cont, cond_cat)
assert balance.dtype == torch.float32
assert entropy.dtype == torch.float32
assert weights.dtype == torch.float32
+34
View File
@@ -10,9 +10,11 @@ from giant.analysis import reduce as R
from giant.analysis.sources import ( from giant.analysis.sources import (
SYNTHETIC_TERMINATION_REASONS, SYNTHETIC_TERMINATION_REASONS,
Side, Side,
open_side,
physical_steps, physical_steps,
secondaries, secondaries,
) )
from giant.data.loader import EVENT_ID_FILE_STRIDE
def _rollout_frame() -> pl.LazyFrame: def _rollout_frame() -> pl.LazyFrame:
@@ -194,3 +196,35 @@ def test_pdg_and_material_labels():
assert G.pdg_label(22) == "gamma" assert G.pdg_label(22) == "gamma"
assert G.pdg_label(999999) == "999999" assert G.pdg_label(999999) == "999999"
assert G.material_label("G4_PbWO4") == "PbWO4" assert G.material_label("G4_PbWO4") == "PbWO4"
def _write_shard(path, event_ids, edeps):
pl.DataFrame({"event_id": event_ids, "pdg": [11] * len(event_ids), "edep": edeps}).write_parquet(path)
def test_open_side_reference_offsets_event_ids_across_shards(tmp_path):
# Each shard is a separate Geant4 job whose own event_id numbering restarts
# from 0 — a naive multi-shard scan collides on event_id across shards.
_write_shard(tmp_path / "a.parquet", [0, 1], [1.0, 2.0])
_write_shard(tmp_path / "b.parquet", [0, 1], [3.0, 4.0])
df = open_side(tmp_path, Side.reference).sort("event_id").collect()
assert df["event_id"].to_list() == [0, 1, EVENT_ID_FILE_STRIDE, EVENT_ID_FILE_STRIDE + 1]
assert df["edep"].to_list() == [1.0, 2.0, 3.0, 4.0]
assert "__source_path" not in df.columns
def test_open_side_reference_single_file_unchanged(tmp_path):
_write_shard(tmp_path / "only.parquet", [0, 1], [1.0, 2.0])
df = open_side(tmp_path / "only.parquet", Side.reference).sort("event_id").collect()
assert df["event_id"].to_list() == [0, 1]
assert "__source_path" not in df.columns
def test_open_side_reference_manifest(tmp_path):
_write_shard(tmp_path / "a.parquet", [0, 1], [1.0, 2.0])
_write_shard(tmp_path / "b.parquet", [0, 1], [3.0, 4.0])
manifest = tmp_path / "shards.manifest"
manifest.write_text("a.parquet\nb.parquet\n")
df = open_side(manifest, Side.reference).sort("event_id").collect()
assert df["event_id"].to_list() == [0, 1, EVENT_ID_FILE_STRIDE, EVENT_ID_FILE_STRIDE + 1]
assert df["edep"].to_list() == [1.0, 2.0, 3.0, 4.0]
+25
View File
@@ -91,6 +91,31 @@ def test_dry_run_writes_nothing(tmp_path: Path):
assert not out_dir.exists() assert not out_dir.exists()
def test_stage1_init_from_and_freeze_flags_scaffold_a_partial_retrain_config(tmp_path: Path):
"""gitea #42."""
out_dir = tmp_path / "run5"
result = runner.invoke(
app,
[
"new-run",
"--out",
str(out_dir),
"--stage1-init-from",
"ckpt/stage1_good/best.pt",
"--stage1-freeze",
],
)
assert result.exit_code == 0, result.output
with open(out_dir / "config.toml", "rb") as f:
cfg = tomllib.load(f)
assert cfg["stage1_model"]["init_from"] == "ckpt/stage1_good/best.pt"
assert cfg["stage1_model"]["freeze"] is True
assert cfg["stage2_model"]["init_from"] == ""
assert cfg["stage2_model"]["freeze"] is False
def test_force_guard_refuses_to_clobber_existing_checkpoints(tmp_path: Path): def test_force_guard_refuses_to_clobber_existing_checkpoints(tmp_path: Path):
out_dir = tmp_path / "run5" out_dir = tmp_path / "run5"
out_dir.mkdir() out_dir.mkdir()
+27
View File
@@ -97,6 +97,33 @@ def test_wgan_knobs_split_per_stage(monkeypatch, tmp_path):
assert cfg["stage2_model"]["wgan"]["gp_weight"] == 2.5 assert cfg["stage2_model"]["wgan"]["gp_weight"] == 2.5
def test_stage1_init_from_and_freeze_flags_land_in_cfg_and_dont_touch_stage2(monkeypatch, tmp_path):
"""gitea #42: --stage{1,2}-init-from/--stage{1,2}-freeze are stage-scoped
only. --stage1-freeze alone would fail validate_config (freeze requires
init_from or --resume), so both flags are passed together here."""
cfg = _invoke_and_capture_cfg(
monkeypatch,
tmp_path,
["--stage1-init-from", "ckpt/stage1_good/best.pt", "--stage1-freeze"],
)
assert cfg["stage1_model"]["init_from"] == "ckpt/stage1_good/best.pt"
assert cfg["stage1_model"]["freeze"] is True
assert cfg["stage2_model"]["init_from"] == ""
assert cfg["stage2_model"]["freeze"] is False
def test_stage2_init_from_and_freeze_flags_land_in_cfg_and_dont_touch_stage1(monkeypatch, tmp_path):
cfg = _invoke_and_capture_cfg(
monkeypatch,
tmp_path,
["--stage2-init-from", "ckpt/stage2_good/best.pt", "--stage2-freeze"],
)
assert cfg["stage2_model"]["init_from"] == "ckpt/stage2_good/best.pt"
assert cfg["stage2_model"]["freeze"] is True
assert cfg["stage1_model"]["init_from"] == ""
assert cfg["stage1_model"]["freeze"] is False
def test_batch_size_invalid_string_errors(monkeypatch, tmp_path): def test_batch_size_invalid_string_errors(monkeypatch, tmp_path):
monkeypatch.setattr(cli, "run_train_job", lambda *a, **kw: None) monkeypatch.setattr(cli, "run_train_job", lambda *a, **kw: None)
result = runner.invoke( result = runner.invoke(
+189 -4
View File
@@ -97,6 +97,19 @@ def test_trunk_config_defaults_block_conditioning_to_add_for_both_stages():
assert gconfig.DEFAULT_CONFIG["stage2_model"]["trunk"]["block_conditioning"] == "add" assert gconfig.DEFAULT_CONFIG["stage2_model"]["trunk"]["block_conditioning"] == "add"
def test_init_from_freeze_default_to_unset_for_both_stages():
"""gitea #42: a pre-existing config with no init_from/freeze key must
reproduce today's from-scratch, always-training behaviour exactly."""
assert gconfig.Stage1ModelConfig().init_from == ""
assert gconfig.Stage1ModelConfig().freeze is False
assert gconfig.Stage2ModelConfig().init_from == ""
assert gconfig.Stage2ModelConfig().freeze is False
assert gconfig.DEFAULT_CONFIG["stage1_model"]["init_from"] == ""
assert gconfig.DEFAULT_CONFIG["stage1_model"]["freeze"] is False
assert gconfig.DEFAULT_CONFIG["stage2_model"]["init_from"] == ""
assert gconfig.DEFAULT_CONFIG["stage2_model"]["freeze"] is False
def test_heads_config_defaults_reproduce_pre_gitea_36_hardcoded_shape(): def test_heads_config_defaults_reproduce_pre_gitea_36_hardcoded_shape():
"""gitea #36: a pre-existing config with no `heads` key must reproduce """gitea #36: a pre-existing config with no `heads` key must reproduce
today's hardcoded `hidden_dim // 2`, one-hidden-layer architecture today's hardcoded `hidden_dim // 2`, one-hidden-layer architecture
@@ -119,7 +132,15 @@ def test_particle_type_config_n_classes_defaults_to_zero_and_round_trips():
assert gconfig.ParticleTypeConfig().n_classes == 0 assert gconfig.ParticleTypeConfig().n_classes == 0
spec = gconfig.ParticleTypeConfig.from_dict({"n_classes": 32}) spec = gconfig.ParticleTypeConfig.from_dict({"n_classes": 32})
assert spec.n_classes == 32 assert spec.n_classes == 32
assert spec.to_dict()["n_classes"] == 32
def test_particle_type_config_class_weighting_defaults_to_none_and_round_trips():
"""gitea #44: an existing config.toml with no
stage2_model.particle_type.class_weighting key must reproduce the
pre-#44 unweighted-CE behavior exactly."""
assert gconfig.ParticleTypeConfig().class_weighting == "none"
spec = gconfig.ParticleTypeConfig.from_dict({"class_weighting": "inverse_freq"})
assert spec.class_weighting == "inverse_freq"
def test_router_config_extra_round_trips_composed_axis_keys(): def test_router_config_extra_round_trips_composed_axis_keys():
@@ -691,6 +712,42 @@ def test_validate_config_embedding_target_passes_with_embedding_conditioning():
gconfig.validate_config(cfg) # must not raise gconfig.validate_config(cfg) # must not raise
def test_validate_config_bad_class_weighting_rejected():
cfg = _cfg_with(**{"stage2_model.particle_type.class_weighting": "effective_num"})
with pytest.raises(ValueError, match="class_weighting"):
gconfig.validate_config(cfg)
def test_validate_config_class_weighting_requires_onehot_target():
cfg = _cfg_with(
**{
"stage2_model.particle_type.class_weighting": "inverse_freq",
"stage2_model.particle_type.target": "physical",
}
)
with pytest.raises(ValueError, match="onehot"):
gconfig.validate_config(cfg)
def test_validate_config_class_weighting_incompatible_with_wgan_generator():
# stage2_model.generator defaults to "wgan" and particle_type.target
# defaults to "onehot", so only class_weighting needs overriding here.
cfg = _cfg_with(**{"stage2_model.particle_type.class_weighting": "inverse_freq"})
with pytest.raises(ValueError, match="wgan"):
gconfig.validate_config(cfg)
def test_validate_config_class_weighting_passes_with_onehot_and_flow():
cfg = _cfg_with(
**{
"stage2_model.particle_type.class_weighting": "inverse_freq",
"stage2_model.particle_type.target": "onehot",
"stage2_model.generator": "flow",
}
)
gconfig.validate_config(cfg) # must not raise
def test_validate_config_mixed_particle_material_conditioning_is_valid(): def test_validate_config_mixed_particle_material_conditioning_is_valid():
"""The particle and material conditioning axes are configured """The particle and material conditioning axes are configured
independently and may mix freely e.g. material independently and may mix freely e.g. material
@@ -735,6 +792,29 @@ def test_validate_config_tie_to_stage1_requires_stage1_active():
assert "tie_to_stage1" in str(e) assert "tie_to_stage1" in str(e)
@pytest.mark.parametrize("stage_name", ["stage1_model", "stage2_model"])
def test_validate_config_freeze_without_init_from_or_resume_rejected(stage_name):
cfg = _cfg_with(**{f"{stage_name}.freeze": True})
try:
gconfig.validate_config(cfg)
assert False, "expected ValueError"
except ValueError as e:
assert "init_from" in str(e)
assert "--resume" in str(e)
@pytest.mark.parametrize("stage_name", ["stage1_model", "stage2_model"])
def test_validate_config_freeze_with_init_from_passes(stage_name):
cfg = _cfg_with(**{f"{stage_name}.freeze": True, f"{stage_name}.init_from": "ckpt/best.pt"})
gconfig.validate_config(cfg) # must not raise
@pytest.mark.parametrize("stage_name", ["stage1_model", "stage2_model"])
def test_validate_config_freeze_without_init_from_passes_under_resume(stage_name):
cfg = _cfg_with(**{f"{stage_name}.freeze": True})
gconfig.validate_config(cfg, resume=True) # must not raise
def test_validate_config_stop_token_accepted_under_autoregressive(): def test_validate_config_stop_token_accepted_under_autoregressive():
"""DEFAULT_CONFIG's stage2_model.decoder is already "autoregressive" """DEFAULT_CONFIG's stage2_model.decoder is already "autoregressive"
(see test_stage2_model_config_defaults_match_documented_v030_intent), so (see test_stage2_model_config_defaults_match_documented_v030_intent), so
@@ -780,13 +860,98 @@ def test_validate_config_bad_stop_sampling_rejected():
assert "stop_sampling" in str(e) assert "stop_sampling" in str(e)
def test_validate_config_stage1_context_sampled_not_implemented(): def test_validate_config_default_precision_is_fp32():
cfg = _cfg_with(**{"stage2_model.stage1_context": "sampled"}) assert gconfig.DEFAULT_CONFIG["train"]["precision"] == "fp32"
def test_validate_config_bf16_precision_accepted():
cfg = _cfg_with(**{"train.precision": "bf16"})
gconfig.validate_config(cfg) # no raise
@pytest.mark.parametrize("bad", ["fp16", "bogus", ""])
def test_validate_config_bad_precision_rejected(bad):
cfg = _cfg_with(**{"train.precision": bad})
try: try:
gconfig.validate_config(cfg) gconfig.validate_config(cfg)
assert False, "expected ValueError" assert False, "expected ValueError"
except ValueError as e: except ValueError as e:
assert "sampled" in str(e) assert "precision" in str(e)
def test_validate_config_stage1_context_sampled_accepted_with_both_stages_active():
"""gitea #41: 'sampled' is now implemented, so DEFAULT_CONFIG's
stage1_model/stage2_model.active = true (both) must let it through."""
cfg = _cfg_with(**{"stage2_model.stage1_context": "sampled"})
gconfig.validate_config(cfg) # must not raise
def test_validate_config_bad_stage1_context_rejected():
cfg = _cfg_with(**{"stage2_model.stage1_context": "bogus"})
try:
gconfig.validate_config(cfg)
assert False, "expected ValueError"
except ValueError as e:
assert "stage1_context" in str(e)
def test_validate_config_stage1_context_sampled_requires_stage1_active():
cfg = _cfg_with(
**{
"stage2_model.stage1_context": "sampled",
"stage1_model.active": False,
}
)
try:
gconfig.validate_config(cfg)
assert False, "expected ValueError"
except ValueError as e:
assert "sampled" in str(e) and "stage1_model.active" in str(e)
def test_validate_config_stage1_context_sampled_requires_stage2_active():
cfg = _cfg_with(
**{
"stage2_model.stage1_context": "sampled",
"stage2_model.active": False,
}
)
try:
gconfig.validate_config(cfg)
assert False, "expected ValueError"
except ValueError as e:
assert "sampled" in str(e) and "stage2_model.active" in str(e)
@pytest.mark.parametrize("key", ["ctx_p_start", "ctx_p_end"])
@pytest.mark.parametrize("value", [-0.1, 1.1])
def test_validate_config_ctx_p_out_of_range_rejected(key, value):
cfg = _cfg_with(
**{
"stage2_model.stage1_context": "sampled",
f"stage2_model.{key}": value,
}
)
try:
gconfig.validate_config(cfg)
assert False, "expected ValueError"
except ValueError as e:
assert key in str(e)
def test_validate_config_stage1_context_sampled_always_truth_rejected_as_noop():
cfg = _cfg_with(
**{
"stage2_model.stage1_context": "sampled",
"stage2_model.ctx_p_start": 1.0,
"stage2_model.ctx_p_end": 1.0,
}
)
try:
gconfig.validate_config(cfg)
assert False, "expected ValueError"
except ValueError as e:
assert "ctx_p_start" in str(e) and "ctx_p_end" in str(e)
def test_validate_config_n_sec_truth_rejected_for_rollout_capable_checkpoint(): def test_validate_config_n_sec_truth_rejected_for_rollout_capable_checkpoint():
@@ -1075,6 +1240,11 @@ def test_overrides_from_flags_train_block_passthrough():
assert overrides == {"train": {"epochs": 5, "lr": 1e-3}} assert overrides == {"train": {"epochs": 5, "lr": 1e-3}}
def test_overrides_from_flags_precision_passthrough():
overrides = gconfig.overrides_from_flags({"precision": "bf16"})
assert overrides == {"train": {"precision": "bf16"}}
@pytest.mark.parametrize( @pytest.mark.parametrize(
("shorthand", "explicit", "path_key"), ("shorthand", "explicit", "path_key"),
[ [
@@ -1193,6 +1363,21 @@ def test_overrides_from_flags_critic_sizing_is_stage_scoped_only(stage_flag, sta
assert overrides == {stage_model: {"wgan": {path_key: 32}}} assert overrides == {stage_model: {"wgan": {path_key: 32}}}
@pytest.mark.parametrize(
("init_from_flag", "freeze_flag", "stage_model"),
[
("stage1_init_from", "stage1_freeze", "stage1_model"),
("stage2_init_from", "stage2_freeze", "stage2_model"),
],
)
def test_overrides_from_flags_init_from_freeze_is_stage_scoped_only(init_from_flag, freeze_flag, stage_model):
"""gitea #42: no shared alias — a checkpoint has one set of weights per
stage, so "freeze both stages from the same file" has no sensible
meaning."""
overrides = gconfig.overrides_from_flags({init_from_flag: "ckpt/best.pt", freeze_flag: True})
assert overrides == {stage_model: {"init_from": "ckpt/best.pt", "freeze": True}}
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# checkpoint config-mismatch warnings (unchanged surface, still exercised) # checkpoint config-mismatch warnings (unchanged surface, still exercised)
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
+1 -20
View File
@@ -28,6 +28,7 @@ import ast
from pathlib import Path from pathlib import Path
from giant.config import DEFAULT_CONFIG from giant.config import DEFAULT_CONFIG
from giant.config import leaf_paths as _leaf_paths
_REPO_ROOT = Path(__file__).resolve().parents[1] _REPO_ROOT = Path(__file__).resolve().parents[1]
@@ -53,13 +54,6 @@ _EXCLUDED_FILES = ("giant/model/_legacy.py",)
# it. If a key here starts showing up as consumed, the fix landed and this # it. If a key here starts showing up as consumed, the fix landed and this
# entry is stale — see test_known_unused_allow_list_has_no_stale_entries. # entry is stale — see test_known_unused_allow_list_has_no_stale_entries.
_KNOWN_UNUSED = { _KNOWN_UNUSED = {
"stage2_model.stage1_context": (
"issues.md Issue 1 — trainers.py hardcodes stage1_ctx to the "
"ground-truth stage-1 output; 'sampled' is now rejected loudly by "
"validate_config (not silently accepted), but the key still isn't "
"read by any build/train consumer file since only 'truth' can pass "
"validation — see Issue 16 for the real implementation"
),
"stage2_model.autoregressive.order": ( "stage2_model.autoregressive.order": (
"gitea #30 — validate_config now checks order is 'energy_desc', but " "gitea #30 — validate_config now checks order is 'energy_desc', but "
"nothing in the build/train/rollout consumer whitelist reads the " "nothing in the build/train/rollout consumer whitelist reads the "
@@ -72,19 +66,6 @@ _KNOWN_UNUSED = {
_FIELD_NAME_OVERRIDES = {"lambda": "lambda_weight"} _FIELD_NAME_OVERRIDES = {"lambda": "lambda_weight"}
def _leaf_paths(node: dict, prefix: str = "") -> list[str]:
paths = []
for key, value in node.items():
if prefix == "" and key == "meta":
continue
path = f"{prefix}.{key}" if prefix else key
if isinstance(value, dict):
paths.extend(_leaf_paths(value, path))
else:
paths.append(path)
return paths
def _field_name(leaf_path: str) -> str: def _field_name(leaf_path: str) -> str:
name = leaf_path.rsplit(".", 1)[-1] name = leaf_path.rsplit(".", 1)[-1]
return _FIELD_NAME_OVERRIDES.get(name, name) return _FIELD_NAME_OVERRIDES.get(name, name)
+7
View File
@@ -195,6 +195,10 @@ def test_build_topn_map_from_files_keeps_most_frequent(tmp_path):
assert m.class_map["G4_Fe"] == 2 # "other" (n_classes - 1) assert m.class_map["G4_Fe"] == 2 # "other" (n_classes - 1)
assert m.class_map["G4_Pb"] == 2 assert m.class_map["G4_Pb"] == 2
assert m.other_members == {"G4_Fe": 2, "G4_Pb": 1} assert m.other_members == {"G4_Fe": 2, "G4_Pb": 1}
# class_counts (gitea #44): per resulting index, "other" is the sum of
# everything folded into it (2 + 1 = 3), and the total equals row count.
assert m.class_counts == {0: 5, 1: 3, 2: 3}
assert sum(m.class_counts.values()) == len(materials)
def test_build_topn_map_from_files_fewer_values_than_n_classes(tmp_path): def test_build_topn_map_from_files_fewer_values_than_n_classes(tmp_path):
@@ -205,6 +209,8 @@ def test_build_topn_map_from_files_fewer_values_than_n_classes(tmp_path):
assert m.class_map == {"G4_AIR": 0, "PbWO4": 1} assert m.class_map == {"G4_AIR": 0, "PbWO4": 1}
assert m.other_members == {} assert m.other_members == {}
# No "other" bucket ever populated -> no entry for its index either.
assert m.class_counts == {0: 1, 1: 1}
def test_build_pdg_topn_map_from_files_pools_primary_and_secondary_pdg(tmp_path): def test_build_pdg_topn_map_from_files_pools_primary_and_secondary_pdg(tmp_path):
@@ -224,6 +230,7 @@ def test_build_pdg_topn_map_from_files_pools_primary_and_secondary_pdg(tmp_path)
# pooled: 11 -> 5, 22 -> 1 (primary) + 10 (secondary) = 11 # pooled: 11 -> 5, 22 -> 1 (primary) + 10 (secondary) = 11
assert m.class_map[22] == 0 assert m.class_map[22] == 0
assert m.class_map[11] == 1 assert m.class_map[11] == 1
assert m.class_counts == {0: 11, 1: 5}
def test_build_pdg_topn_map_from_files_missing_sec_pdg_list_column(tmp_path): def test_build_pdg_topn_map_from_files_missing_sec_pdg_list_column(tmp_path):
+137
View File
@@ -0,0 +1,137 @@
"""Tests for `giant model summary` (gitea #46)."""
from __future__ import annotations
from pathlib import Path
import pytest
from typer.testing import CliRunner
from giant import config as gconfig
from giant.cli import app
from giant.materials import MATERIAL_PROPERTIES
from giant.model.summary import _NOT_BUILD_TIME, _built_modules, _vocab_caveats, summarize_model
runner = CliRunner()
_PDG_VOCAB = 300
_MAT_VOCAB = len(MATERIAL_PROPERTIES)
def _cfg(overrides: dict | None = None) -> dict:
return gconfig.merge_cli_overrides(gconfig.DEFAULT_CONFIG, None, overrides or {})
@pytest.fixture(scope="module")
def default_summary():
return summarize_model(_cfg(), pdg_vocab=_PDG_VOCAB, mat_vocab=_MAT_VOCAB)
def test_default_config_builds_both_stages_with_a_real_tree(default_summary):
assert set(default_summary.modules) >= {"stage1", "stage2"}
for module in default_summary.modules.values():
assert sum(p.numel() for p in module.parameters()) > 0
stage1 = default_summary.modules["stage1"]
assert hasattr(stage1, "cond_enc")
assert hasattr(stage1, "trunk")
assert {"input_proj", "blocks", "out_proj"} <= {n for n, _ in stage1.trunk.named_children()}
def test_every_in_scope_leaf_is_classified(default_summary):
in_scope = {
p
for p in gconfig.leaf_paths(gconfig.DEFAULT_CONFIG)
if p.split(".", 1)[0] in ("conditioning", "stage1_model", "stage2_model")
}
classified = set(default_summary.consumed) | set(default_summary.inert) | set(default_summary.elsewhere)
assert classified == in_scope
def test_not_build_time_allow_list_has_no_stale_entries():
in_scope = set(gconfig.leaf_paths(gconfig.DEFAULT_CONFIG))
stale = set(_NOT_BUILD_TIME) - in_scope
assert not stale, f"_NOT_BUILD_TIME entries no longer in DEFAULT_CONFIG: {sorted(stale)}"
def test_router_disabled_by_default_so_its_fields_are_inert(default_summary):
assert "stage1_model.router.n_experts" in default_summary.inert
assert "stage1_model.router.temperature" in default_summary.inert
def test_markov_history_leaves_attention_dims_inert_but_history_itself_consumed(default_summary):
assert "stage2_model.autoregressive.attn_n_heads" in default_summary.inert
assert "stage2_model.autoregressive.attn_n_layers" in default_summary.inert
assert "stage2_model.autoregressive.history" in default_summary.consumed
def test_single_literal_branch_fields_are_correctly_seen_as_consumed(default_summary):
"""Regression guard: n_sec.owner ("stage2"), n_sec.mode ("head") and
particle_type.target ("onehot") each branch as `== "one specific other
literal"` in giant/model/builders.py|models.py. A naive single generic
sentinel probe lands in the same "not that literal" bucket as the
current value and never crosses the boundary that actually matters --
this is exactly what _STRING_ALTERNATIVES exists to fix."""
assert "stage2_model.n_sec.owner" in default_summary.consumed
assert "stage2_model.n_sec.mode" in default_summary.consumed
assert "stage2_model.particle_type.target" in default_summary.consumed
def test_stage1_wgan_generator_swaps_flow_time_dim_for_critic_dims():
summary = summarize_model(_cfg({"stage1_model": {"generator": "wgan"}}), pdg_vocab=_PDG_VOCAB, mat_vocab=_MAT_VOCAB)
assert "stage1_model.flow.time_dim" in summary.inert
assert "stage1_model.wgan.noise_dim" in summary.consumed
assert "stage1_model.wgan.critic_hidden_dim" in summary.consumed
def test_stage2_one_shot_decoder_makes_autoregressive_block_inert():
summary = summarize_model(
_cfg({"stage2_model": {"decoder": "one_shot"}}), pdg_vocab=_PDG_VOCAB, mat_vocab=_MAT_VOCAB
)
assert "stage2_model.autoregressive.history" in summary.inert
assert "history_encoder" not in {n for n, _ in summary.modules["stage2"].named_children()}
def test_energy_router_enabled_consumes_core_fields_but_not_process_only_fields():
summary = summarize_model(
_cfg({"stage1_model": {"router": {"enabled": True, "type": "energy", "n_experts": 4}}}),
pdg_vocab=_PDG_VOCAB,
mat_vocab=_MAT_VOCAB,
)
assert "stage1_model.router.n_experts" in summary.consumed
assert "stage1_model.router.temperature" in summary.consumed
# emb_dim/hidden_dim are pdg/process-router-only kwargs -- build_router's
# signature filter drops them for an energy router.
assert "stage1_model.router.hidden_dim" in summary.inert
assert "stage1_model.router.emb_dim" in summary.inert
def test_vocab_caveat_text_for_embedding_particle_conditioning():
cfg = _cfg({"conditioning": {"particle": {"type": "embedding"}}})
caveats = _vocab_caveats(cfg)
assert any("pdg_vocab" in c and "embedding" in c for c in caveats)
assert not any("mat_vocab" in c for c in caveats)
def test_pdg_vocab_flag_changes_embedding_table_size():
cfg = _cfg({"conditioning": {"particle": {"type": "embedding"}}})
small = _built_modules(cfg, pdg_vocab=10, mat_vocab=_MAT_VOCAB)
big = _built_modules(cfg, pdg_vocab=1000, mat_vocab=_MAT_VOCAB)
assert big["stage1"].cond_enc.pdg_emb.weight.numel() > small["stage1"].cond_enc.pdg_emb.weight.numel()
def test_invalid_combo_exits_nonzero_with_validate_config_message(tmp_path: Path):
config_path = tmp_path / "bad.toml"
config_path.write_text('[meta]\nconfig_version = 3\n\n[stage2_model.particle_type]\ntarget = "embedding"\n')
result = runner.invoke(app, ["model", "summary", "--config", str(config_path)])
assert result.exit_code == 1
assert "requires conditioning.particle.type = 'embedding'" in result.output
def test_cli_default_smoke():
result = runner.invoke(app, ["model", "summary"])
assert result.exit_code == 0, result.output
assert "stage1" in result.output
assert "stage2" in result.output
assert "parameters" in result.output
assert "trunk" in result.output
assert "inert under this config" in result.output
+40 -3
View File
@@ -10,6 +10,7 @@ from giant.model.network import (
ConditionEncoder, ConditionEncoder,
HistoryEncoder, HistoryEncoder,
MarkovHistory, MarkovHistory,
NoHistory,
SinusoidalEmbedding, SinusoidalEmbedding,
Stage1Model, Stage1Model,
Stage2Autoregressive, Stage2Autoregressive,
@@ -465,16 +466,52 @@ def test_attention_history_step_matches_forward():
assert torch.allclose(stepped, expected, atol=1e-5) assert torch.allclose(stepped, expected, atol=1e-5)
# --- NoHistory (gitea #45) ----------------------------------------------------
def test_no_history_shape():
hist = NoHistory(in_dim=7, out_dim=12)
B, K = 3, 5
feat = torch.randn(B, K, 7)
has_prev = (torch.arange(K) >= 1).unsqueeze(0).expand(B, -1)
out = hist(feat, has_prev)
assert out.shape == (B, K, 12)
def test_no_history_ignores_feat_and_has_prev():
hist = NoHistory(in_dim=4, out_dim=6)
B, K = 2, 3
has_prev_a = (torch.arange(K) >= 1).unsqueeze(0).expand(B, -1)
has_prev_b = torch.zeros(B, K, dtype=torch.bool)
feat_a = torch.randn(B, K, 4)
feat_b = torch.randn(B, K, 4) * 100
out_a = hist(feat_a, has_prev_a)
out_b = hist(feat_b, has_prev_b)
assert torch.equal(out_a, torch.zeros(B, K, 6))
assert torch.equal(out_a, out_b)
def test_no_history_uses_base_class_o1_defaults():
hist = NoHistory(in_dim=4, out_dim=6)
assert hist.init_cache() is None
feat = torch.randn(2, 1, 4)
has_prev = torch.ones(2, 1, dtype=torch.bool)
out, cache = hist.step(feat, has_prev, "unused-cache")
assert torch.equal(out, torch.zeros(2, 1, 6))
assert cache == "unused-cache"
# --- HISTORY_REGISTRY / build_history (gitea #35) ---------------------------- # --- HISTORY_REGISTRY / build_history (gitea #35) ----------------------------
def test_history_registry_has_exactly_the_two_known_histories(): def test_history_registry_has_exactly_the_known_histories():
assert set(HISTORY_REGISTRY) == {"markov", "attention"} assert set(HISTORY_REGISTRY) == {"markov", "attention", "none"}
def test_build_history_returns_correct_concrete_type(): def test_build_history_returns_correct_concrete_type():
assert isinstance(build_history("markov", 4, 6), MarkovHistory) assert isinstance(build_history("markov", 4, 6), MarkovHistory)
assert isinstance(build_history("attention", 4, 8), AttentionHistory) assert isinstance(build_history("attention", 4, 8), AttentionHistory)
assert isinstance(build_history("none", 4, 6), NoHistory)
def test_build_history_unknown_name_raises(): def test_build_history_unknown_name_raises():
@@ -605,7 +642,7 @@ def test_stage2_autoregressive_n_sec_head_and_type_head_cfg_control_hidden_width
@pytest.mark.parametrize("target", ["physical", "onehot", "embedding"]) @pytest.mark.parametrize("target", ["physical", "onehot", "embedding"])
@pytest.mark.parametrize("generator", ["wgan", "flow"]) @pytest.mark.parametrize("generator", ["wgan", "flow"])
@pytest.mark.parametrize("history", ["markov", "attention"]) @pytest.mark.parametrize("history", ["markov", "attention", "none"])
def test_stage2_autoregressive_forward_shape(target, generator, history): def test_stage2_autoregressive_forward_shape(target, generator, history):
B, K, emb_dim = 4, 5, 6 B, K, emb_dim = 4, 5, 6
model = _build_stage2_ar(target, generator, emb_dim=emb_dim, k_max=K, history=history) model = _build_stage2_ar(target, generator, emb_dim=emb_dim, k_max=K, history=history)
+1 -1
View File
@@ -231,7 +231,7 @@ def test_run_train_job_builds_caches_and_persists_material_topn_map(tmp_path, da
def test_run_train_job_no_topn_map_for_physical_target(tmp_path, data): def test_run_train_job_no_topn_map_for_physical_target(tmp_path, data):
cfg = _tiny_cfg() cfg = _tiny_cfg()
cfg["stage2_model"]["particle_type"] = {"target": "physical", "lambda": 1.0} cfg["stage2_model"]["particle_type"].update({"target": "physical", "lambda": 1.0})
echo = _run(data, tmp_path / "out", cfg=cfg) echo = _run(data, tmp_path / "out", cfg=cfg)
assert not any("top-N map" in m for m in echo) assert not any("top-N map" in m for m in echo)
+77
View File
@@ -0,0 +1,77 @@
"""Config-correctness tests for the CI version-bump/tag/changelog automation
(gitea #50). The workflow YAML itself can only be exercised by a real push to
master, so these check the two config files it drives (.bumpversion.toml,
cliff.toml) against real repo content instead.
"""
import re
import shutil
import subprocess
import tomllib
from pathlib import Path
import pytest
_ROOT = Path(__file__).resolve().parents[1]
def test_bumpversion_search_pattern_matches_pyproject():
bump_config = tomllib.loads((_ROOT / ".bumpversion.toml").read_text())["tool"]["bumpversion"]
current_version = bump_config["current_version"]
search = bump_config["files"][0]["search"].format(current_version=current_version)
pyproject = (_ROOT / "pyproject.toml").read_text()
assert search in pyproject, (
f"bumpversion search pattern {search!r} (rendered from .bumpversion.toml's "
f"current_version={current_version!r}) not found in pyproject.toml — "
"the bump would silently edit nothing"
)
@pytest.mark.skipif(shutil.which("git-cliff") is None, reason="git-cliff binary not on PATH")
def test_cliff_config_groups_and_links_commits(tmp_path):
repo = tmp_path / "repo"
repo.mkdir()
subprocess.run(["git", "init", "-q"], cwd=repo, check=True)
subprocess.run(["git", "config", "user.name", "test"], cwd=repo, check=True)
subprocess.run(["git", "config", "user.email", "test@example.com"], cwd=repo, check=True)
_commit(repo, "Add class-balanced secondary particle-type loss (gitea #44)")
_commit(repo, "Fix leaking secondary energy budget")
_commit(repo, "Merge pull request 'Add X' (#1) from fix/issue-1 into master")
_commit(repo, "chore: bump version 0.3.3 -> 0.3.4 [skip ci]")
result = subprocess.run(
[
"git-cliff",
"--config",
str(_ROOT / "cliff.toml"),
"--repository",
str(repo),
"--tag",
"v0.3.4",
"--unreleased",
],
capture_output=True,
text=True,
check=True,
)
changelog = result.stdout
assert "## [0.3.4]" in changelog
assert "### Added" in changelog
assert "### Fixed" in changelog
assert re.search(
r"\[gitea #44\]\(https://git\.larsbogner\.de/lars/giant/issues/44\)",
changelog,
)
assert "Add class-balanced secondary particle-type loss" in changelog
assert "Fix leaking secondary energy budget" in changelog
assert "Merge pull request" not in changelog
assert "skip ci" not in changelog
def _commit(repo: Path, message: str) -> None:
(repo / "f.txt").write_text(message)
subprocess.run(["git", "add", "f.txt"], cwd=repo, check=True)
subprocess.run(["git", "commit", "-q", "-m", message], cwd=repo, check=True)
+105
View File
@@ -13,6 +13,8 @@ from giant.model.network import (
EnergyRouter, EnergyRouter,
ExpertTrunk, ExpertTrunk,
FilmResBlock, FilmResBlock,
LinearTrunk,
NoneRouter,
PdgRouter, PdgRouter,
ProcessRouter, ProcessRouter,
ROUTER_REGISTRY, ROUTER_REGISTRY,
@@ -73,6 +75,34 @@ def test_energy_router_registered():
assert ROUTER_REGISTRY["energy"] is EnergyRouter assert ROUTER_REGISTRY["energy"] is EnergyRouter
# ── NoneRouter (gitea #45) ──────────────────────────────────────────────────
def test_none_router_registered():
assert ROUTER_REGISTRY["none"] is NoneRouter
def test_none_router_gate_is_uniform():
router = NoneRouter(n_experts=4)
cond_cont, cond_cat = _cond(16)
g = router.gate(cond_cont, cond_cat)
assert g.shape == (16, 4)
torch.testing.assert_close(g, torch.full((16, 4), 0.25))
def test_none_router_gate_ignores_conditioning():
router = NoneRouter(n_experts=3)
cond_cont_a, cond_cat_a = _cond(8)
cond_cont_b, cond_cat_b = _cond(8)
torch.testing.assert_close(router.gate(cond_cont_a, cond_cat_a), router.gate(cond_cont_b, cond_cat_b))
def test_none_router_top1_always_expert_zero():
router = NoneRouter(n_experts=4)
cond_cont, cond_cat = _cond(16)
assert torch.equal(router.top1(cond_cont, cond_cat), torch.zeros(16, dtype=torch.long))
def test_energy_router_gate_partition_of_unity(): def test_energy_router_gate_partition_of_unity():
router = EnergyRouter(n_experts=4) router = EnergyRouter(n_experts=4)
cond_cont, cond_cat = _cond(16) cond_cont, cond_cat = _cond(16)
@@ -168,6 +198,47 @@ def test_build_expert_body_unknown_type_raises():
raise AssertionError("expected ValueError for unknown trunk type") raise AssertionError("expected ValueError for unknown trunk type")
# ── LinearTrunk (gitea #45) ──────────────────────────────────────────────────
def test_trunk_registry_has_linear():
assert "linear" in TRUNK_REGISTRY
assert TRUNK_REGISTRY["linear"] is LinearTrunk
def test_linear_trunk_forward_shape():
trunk = build_expert_body("linear", in_dim=9, out_dim=9, hidden_dim=64, n_blocks=6, cond_dim=12)
assert trunk.in_dim == 9
assert trunk.out_dim == 9
x = torch.randn(5, 9)
cond = torch.randn(5, 12)
out = trunk(x, cond)
assert out.shape == (5, 9)
def test_linear_trunk_depends_on_x_and_cond():
trunk = build_expert_body("linear", in_dim=9, out_dim=9, hidden_dim=64, n_blocks=6, cond_dim=12)
x = torch.randn(5, 9)
cond_a = torch.randn(5, 12)
cond_b = torch.randn(5, 12)
assert not torch.allclose(trunk(x, cond_a), trunk(x, cond_b))
def test_routed_linear_trunk_is_mixture_of_trivial_experts():
"""trunk.type = 'linear' composes for free with router.enabled = true
(gitea #33's comment on this issue) — a RoutedTrunk of n_experts linear
bodies."""
router = build_router("energy", n_experts=3)
trunk = RoutedTrunk(router, "linear", in_dim=9, out_dim=9, hidden_dim=64, n_res_blocks=6, cond_dim=12)
assert len(trunk.experts) == 3
assert all(isinstance(e, LinearTrunk) for e in trunk.experts)
x = torch.randn(5, 9)
cond = torch.randn(5, 12)
cond_cont, cond_cat = _cond(5)
out = trunk(x, cond, cond_cont, cond_cat)
assert out.shape == (5, 9)
# ── BLOCK_REGISTRY / build_block (gitea #34) ──────────────────────────────── # ── BLOCK_REGISTRY / build_block (gitea #34) ────────────────────────────────
@@ -1195,3 +1266,37 @@ def test_build_models_routed_pair_is_drop_in_for_sample_flow():
) )
assert sec_cont.shape == (B, K_MAX, 4) assert sec_cont.shape == (B, K_MAX, 4)
assert sec_valid.shape == (B, K_MAX) assert sec_valid.shape == (B, K_MAX)
def test_routed_and_unrouted_trunk_agree_on_dtype_under_bf16_autocast():
"""gitea #47 regression: `_route_forward`'s accumulator (giant/model/
trunks.py) used to be a hard-fp32 `torch.zeros`, so under autocast a
`RoutedTrunk` returned fp32 while an unrouted `ExpertTrunk` returned
bf16 `router.enabled` alone silently changed the model's output dtype.
Checked in both train mode (the differentiable mixture sum) and eval
mode (the masked `out[mask] = expert(...)` dispatch) the two branches
of `_route_forward` had independent copies of the bug."""
torch.manual_seed(0)
cond_cont, cond_cat = _cond(B=6)
x = torch.randn(6, X_DIM)
t = torch.rand(6)
unrouted = Stage1Model(
pdg_vocab=3,
mat_vocab=2,
particle_cfg=PARTICLE_CFG,
material_cfg=MATERIAL_CFG,
hidden_dim=16,
n_res_blocks=2,
)
routed = _routed_stage1(n_experts=3)
for train_mode in (True, False):
unrouted.train(train_mode)
routed.train(train_mode)
with torch.autocast("cpu", dtype=torch.bfloat16, enabled=True):
out_unrouted = unrouted(x, cond_cont, cond_cat, t=t)
out_routed = routed(x, cond_cont, cond_cat, t=t)
assert out_unrouted.dtype == out_routed.dtype, (
f"train={train_mode}: unrouted returned {out_unrouted.dtype}, routed returned {out_routed.dtype}"
)
+13 -1
View File
@@ -99,7 +99,7 @@ def test_save_load_round_trip_topn_maps(tmp_path):
cache = SetupCache.empty(files) cache = SetupCache.empty(files)
cache.topn_maps[setup_cache.topn_key("pdg", 3)] = TopNMap( cache.topn_maps[setup_cache.topn_key("pdg", 3)] = TopNMap(
class_map={22: 0, 11: 1, 2212: 2}, other_members={2212: 5} class_map={22: 0, 11: 1, 2212: 2}, other_members={2212: 5}, class_counts={0: 100, 1: 50, 2: 5}
) )
cache.topn_maps[setup_cache.topn_key("material", 2)] = TopNMap( cache.topn_maps[setup_cache.topn_key("material", 2)] = TopNMap(
class_map={"G4_AIR": 0, "PbWO4": 1}, other_members={} class_map={"G4_AIR": 0, "PbWO4": 1}, other_members={}
@@ -114,9 +114,21 @@ def test_save_load_round_trip_topn_maps(tmp_path):
assert pdg_m.other_members == {2212: 5} assert pdg_m.other_members == {2212: 5}
# key type is int (matches pdg_map's own key type), not str # key type is int (matches pdg_map's own key type), not str
assert all(isinstance(k, int) for k in pdg_m.class_map) assert all(isinstance(k, int) for k in pdg_m.class_map)
# class_counts (gitea #44) round-trips too, keyed by class index (always
# int, independent of the pdg/material axis's own key type).
assert pdg_m.class_counts == {0: 100, 1: 50, 2: 5}
assert all(isinstance(k, int) for k in pdg_m.class_counts)
mat_m = loaded.topn_maps[setup_cache.topn_key("material", 2)] mat_m = loaded.topn_maps[setup_cache.topn_key("material", 2)]
assert mat_m.class_map == {"G4_AIR": 0, "PbWO4": 1} assert mat_m.class_map == {"G4_AIR": 0, "PbWO4": 1}
assert mat_m.class_counts == {}
def test_topnmap_from_json_missing_class_counts_defaults_empty():
"""A checkpoint's topn map predating gitea #44 has no class_counts key at
all must decode to {}, not raise, since inference never reads it."""
m = setup_cache.topnmap_from_json({"class_map": {"11": 0}, "other_members": {}}, axis="pdg")
assert m.class_counts == {}
def test_topn_key_unknown_axis_raises(): def test_topn_key_unknown_axis_raises():
+442 -3
View File
@@ -5,10 +5,12 @@ import csv
import math import math
import tempfile import tempfile
from pathlib import Path from pathlib import Path
from unittest.mock import MagicMock from unittest.mock import MagicMock, patch
import numpy as np
import pytest import pytest
import torch import torch
import torch.nn.functional as F
from giant.config import ParticleTypeConfig from giant.config import ParticleTypeConfig
from giant.constants import ( from giant.constants import (
@@ -19,16 +21,22 @@ from giant.constants import (
SEC_SLOT_DIM, SEC_SLOT_DIM,
X_DIM, X_DIM,
) )
from giant.checkpoint_io import load_for_inference
from giant.data.dataset import StepBatch from giant.data.dataset import StepBatch
from giant.data.transforms import Normalizer
from giant.model.network import Stage2Autoregressive, build_critics, build_models from giant.model.network import Stage2Autoregressive, build_critics, build_models
from giant.sample import sample_stage1 as trainers_sample_stage1
from giant.training import ( from giant.training import (
FlowDDPMStageTrainer, FlowDDPMStageTrainer,
StageSpec, StageSpec,
WGANStageTrainer, WGANStageTrainer,
build_checkpoint,
build_stage_trainers, build_stage_trainers,
init_stages_from_checkpoints,
train, train,
) )
from giant.training.metrics import _wandb_run_config from giant.training.metrics import _wandb_run_config
from giant.training.trainers import _type_class_weight_vector
from giant.training.stage2_inputs import ( from giant.training.stage2_inputs import (
_ar_has_prev, _ar_has_prev,
_assemble_stage2_ar_inputs, _assemble_stage2_ar_inputs,
@@ -351,7 +359,7 @@ def _model_config(cfg):
} }
def _run_train(cfg, out_dir, resume_path=None): def _run_train(cfg, out_dir, resume_path=None, normalizer_dict=None):
model_config = _model_config(cfg) model_config = _model_config(cfg)
models = build_models(model_config) models = build_models(model_config)
critics = build_critics(model_config) critics = build_critics(model_config)
@@ -365,7 +373,7 @@ def _run_train(cfg, out_dir, resume_path=None):
val_loader=val_loader, val_loader=val_loader,
device=torch.device("cpu"), device=torch.device("cpu"),
out_dir=out_dir, out_dir=out_dir,
normalizer_dict={"cond": {}, "target": {}, "sec_phys": {}}, normalizer_dict=normalizer_dict or {"cond": {}, "target": {}, "sec_phys": {}},
pdg_map={"22": 0}, pdg_map={"22": 0},
mat_map={"G4_AIR": 0}, mat_map={"G4_AIR": 0},
proc_map=None, proc_map=None,
@@ -593,6 +601,313 @@ def test_flow_stage_trainer_ddpm_not_implemented_for_stage2():
FlowDDPMStageTrainer(spec, torch.nn.Linear(1, 1), torch.device("cpu")) FlowDDPMStageTrainer(spec, torch.nn.Linear(1, 1), torch.device("cpu"))
# --- gitea #44: class-balanced secondary particle-type loss -----------------
def test_type_class_weight_vector_none_scheme_returns_none():
assert _type_class_weight_vector({0: 100, 1: 5}, n_classes=2, scheme="none") is None
def test_type_class_weight_vector_raises_without_counts():
with pytest.raises(ValueError, match="class_counts"):
_type_class_weight_vector({}, n_classes=4, scheme="inverse_freq")
def test_type_class_weight_vector_inverse_freq_favors_rare_class_and_has_mean_one():
weights = _type_class_weight_vector({0: 1000, 1: 10, 2: 1, 3: 1}, n_classes=4, scheme="inverse_freq")
assert weights is not None
assert len(weights) == 4
assert weights[1] > weights[0] # rarer class -> larger weight
assert math.isclose(sum(weights) / len(weights), 1.0, rel_tol=1e-9)
def test_type_class_weight_vector_missing_index_clamps_to_count_one():
# n_classes=3 but only index 0 was ever observed (e.g. a tiny dataset) —
# indices 1/2 must not divide by zero.
weights = _type_class_weight_vector({0: 10}, n_classes=3, scheme="inverse_freq")
assert weights is not None
assert all(math.isfinite(w) for w in weights)
def _onehot_flow_stage2_setup():
"""A built stage-2 model + a batch, under target='onehot' + generator='flow'
(mirrors the 'stage2_onehot_target_flow' case in test_train_end_to_end)."""
cfg = _base_cfg()
cfg["stage2_model"]["generator"] = "flow"
cfg["stage2_model"]["particle_type"] = {"target": "onehot", "lambda": 1.0}
model_config = _model_config(cfg)
model = build_models(model_config)["stage2"]
assert model is not None
batch = _fake_batches(1, 8)[0]
device = torch.device("cpu")
cond_cont, cond_cat, x1_s1 = batch.cond_cont, batch.cond_cat, batch.target_s1
# Mostly class 0 (common), a few slot 1's set to class 1 (rare) —
# PARTICLE_CFG's emb_dim=8, n_classes=0 (inherit) -> 8 type classes.
sec_type_idx = torch.zeros(8, K_MAX, dtype=torch.long)
sec_type_idx[:, :2] = 1
sec_mask = torch.ones(8, K_MAX, dtype=torch.bool)
return model, cond_cont, cond_cat, x1_s1, sec_type_idx, sec_mask, device
def test_flow_ddpm_trainer_type_loss_none_leaves_weight_unset():
model, *_ = _onehot_flow_stage2_setup()
spec = StageSpec(
name="stage2",
is_stage2=True,
generator="flow",
particle_type=ParticleTypeConfig(target="onehot", class_weighting="none"),
particle_type_n_classes=8,
ema_decay=0.0,
)
trainer = FlowDDPMStageTrainer(spec, model, torch.device("cpu"))
assert trainer.type_class_weights is None
def test_flow_ddpm_trainer_type_loss_matches_manual_weighted_cross_entropy():
model, cond_cont, cond_cat, x1_s1, sec_type_idx, sec_mask, device = _onehot_flow_stage2_setup()
class_counts = {0: 1000, 1: 10, 2: 1, 3: 1, 4: 1, 5: 1, 6: 1, 7: 1}
weights = _type_class_weight_vector(class_counts, n_classes=8, scheme="inverse_freq")
spec = StageSpec(
name="stage2",
is_stage2=True,
generator="flow",
particle_type=ParticleTypeConfig(target="onehot", class_weighting="inverse_freq"),
particle_type_n_classes=8,
type_class_weights=weights,
ema_decay=0.0,
)
trainer = FlowDDPMStageTrainer(spec, model, device)
assert trainer.type_class_weights is not None
stage1_ctx = trainer._stage1_context(x1_s1, cond_cont, cond_cat, epoch=None)
with torch.no_grad():
type_out = model.predict_type(cond_cont, cond_cat, stage1_ctx)
weight_t = torch.tensor(weights)
ce = F.cross_entropy(type_out.transpose(1, 2), sec_type_idx, weight=weight_t, reduction="none")
expected = (ce * sec_mask.float()).sum() / sec_mask.float().sum().clamp(min=1)
l_type, _ = trainer._type_loss(cond_cont, cond_cat, stage1_ctx, sec_type_idx, sec_mask, device)
assert torch.allclose(l_type, expected, atol=1e-6)
# Unweighted trainer, same model/batch — the two losses must differ
# (the batch mixes the common and rare classes, so weighting changes the
# per-slot contributions), confirming the weight is actually plumbed in.
spec_none = StageSpec(
name="stage2",
is_stage2=True,
generator="flow",
particle_type=ParticleTypeConfig(target="onehot", class_weighting="none"),
particle_type_n_classes=8,
ema_decay=0.0,
)
trainer_none = FlowDDPMStageTrainer(spec_none, model, device)
with torch.no_grad():
l_type_none, _ = trainer_none._type_loss(cond_cont, cond_cat, stage1_ctx, sec_type_idx, sec_mask, device)
assert not torch.allclose(l_type, l_type_none)
def test_build_stage_trainers_threads_sec_type_class_counts_into_weights():
cfg = _base_cfg()
cfg["stage2_model"]["generator"] = "flow"
cfg["stage2_model"]["particle_type"] = {
"target": "onehot",
"lambda": 1.0,
"class_weighting": "inverse_freq",
}
model_config = _model_config(cfg)
models = build_models(model_config)
critics = build_critics(model_config)
class_counts = {i: 100 for i in range(8)}
class_counts[1] = 1 # one rare class
trainers = build_stage_trainers(
cfg, models, critics, torch.device("cpu"), total_train_batches=4, sec_type_class_counts=class_counts
)
stage2_trainer = trainers["stage2"]
assert isinstance(stage2_trainer, FlowDDPMStageTrainer)
weights = stage2_trainer.type_class_weights
assert weights is not None
assert weights[1] > weights[0]
def test_build_stage_trainers_no_class_counts_with_none_weighting_is_fine():
"""The overwhelmingly common case (class_weighting = 'none', the
default): build_stage_trainers must not require sec_type_class_counts at
all."""
cfg = _base_cfg()
cfg["stage2_model"]["generator"] = "flow"
cfg["stage2_model"].__setitem__("particle_type", {"target": "onehot", "lambda": 1.0})
model_config = _model_config(cfg)
models = build_models(model_config)
critics = build_critics(model_config)
trainers = build_stage_trainers(cfg, models, critics, torch.device("cpu"), total_train_batches=4)
stage2_trainer = trainers["stage2"]
assert isinstance(stage2_trainer, FlowDDPMStageTrainer)
assert stage2_trainer.type_class_weights is None
# --- gitea #42: freeze / init_from -------------------------------------------
def _state_dict_clone(module):
return {k: v.clone() for k, v in module.state_dict().items()}
def _assert_state_dicts_equal(before, after, label):
for key, value in before.items():
assert torch.equal(value, after[key]), f"{label}: {key} changed while frozen"
def test_frozen_flow_stage_trainer_step_does_not_update_model_or_ema():
cfg = _base_cfg()
model_config = _model_config(cfg)
model = build_models(model_config)["stage1"]
assert model is not None
spec = StageSpec(name="stage1", is_stage2=False, generator="flow", freeze=True, ema_decay=0.999, steps_per_epoch=4)
trainer = FlowDDPMStageTrainer(spec, model, torch.device("cpu"))
assert trainer.ema_model is not None
model_before = _state_dict_clone(trainer.model)
ema_before = _state_dict_clone(trainer.ema_model)
for batch in _fake_batches(4, 8):
trainer.step(batch, torch.device("cpu"), global_step=1)
_assert_state_dicts_equal(model_before, trainer.model.state_dict(), "frozen flow model")
_assert_state_dicts_equal(ema_before, trainer.ema_model.state_dict(), "frozen flow ema")
def test_frozen_wgan_stage_trainer_step_does_not_update_generator_or_critic():
cfg = _base_cfg()
cfg["stage1_model"]["generator"] = "wgan"
model_config = _model_config(cfg)
models = build_models(model_config)
critics = build_critics(model_config)
assert models["stage1"] is not None and critics["stage1"] is not None
spec = StageSpec(
name="stage1",
is_stage2=False,
generator="wgan",
freeze=True,
n_critic=1, # a generator step every batch, so a bug would surface immediately
ema_decay=0.999,
steps_per_epoch=4,
)
trainer = WGANStageTrainer(spec, models["stage1"], critics["stage1"], torch.device("cpu"))
assert trainer.ema_model is not None
model_before = _state_dict_clone(trainer.model)
critic_before = _state_dict_clone(trainer.critic)
ema_before = _state_dict_clone(trainer.ema_model)
for global_step, batch in enumerate(_fake_batches(4, 8)):
trainer.step(batch, torch.device("cpu"), global_step=global_step)
_assert_state_dicts_equal(model_before, trainer.model.state_dict(), "frozen wgan generator")
_assert_state_dicts_equal(critic_before, trainer.critic.state_dict(), "frozen wgan critic")
_assert_state_dicts_equal(ema_before, trainer.ema_model.state_dict(), "frozen wgan ema")
@pytest.mark.parametrize("stage1_generator", ["flow", "wgan"])
def test_train_end_to_end_frozen_stage1_unchanged_while_stage2_trains(stage1_generator):
cfg = _base_cfg()
cfg["stage1_model"]["generator"] = stage1_generator
cfg["stage1_model"]["freeze"] = True
model_config = _model_config(cfg)
models = build_models(model_config)
critics = build_critics(model_config)
assert models["stage1"] is not None and models["stage2"] is not None
stage1_before = _state_dict_clone(models["stage1"])
stage2_before = _state_dict_clone(models["stage2"])
with tempfile.TemporaryDirectory() as tmp:
train(
cfg=cfg,
models=models,
critics=critics,
train_loader=_fake_batches(4, cfg["train"]["batch_size"]),
val_loader=_fake_batches(2, cfg["train"]["batch_size"], seed=1),
device=torch.device("cpu"),
out_dir=Path(tmp) / "run",
normalizer_dict={"cond": {}, "target": {}, "sec_phys": {}},
pdg_map={"22": 0},
mat_map={"G4_AIR": 0},
proc_map=None,
model_config=model_config,
total_train_batches=4,
)
_assert_state_dicts_equal(stage1_before, models["stage1"].state_dict(), "frozen stage1")
stage2_after = models["stage2"].state_dict()
assert any(not torch.equal(v, stage2_after[k]) for k, v in stage2_before.items()), (
"unfrozen stage2 should have trained"
)
def test_init_stages_from_checkpoints_loads_matching_stage_and_ema_weights(tmp_path):
cfg = _base_cfg()
model_config = _model_config(cfg)
source_models = build_models(model_config)
source_critics = build_critics(model_config)
source_trainers = build_stage_trainers(cfg, source_models, source_critics, torch.device("cpu"), 4)
source_stage1_ema = source_trainers["stage1"].ema_model
assert source_stage1_ema is not None
# Diverge the source's EMA from its raw weights so a same-vs-different
# check below actually distinguishes the two copy paths.
for p in source_stage1_ema.parameters():
p.data.add_(1.0)
ckpt_path = tmp_path / "source.pt"
ckpt = build_checkpoint(source_trainers, epoch=1, global_step=1, best_val_loss=0.0, extras={})
torch.save(ckpt, ckpt_path)
cfg2 = copy.deepcopy(cfg)
cfg2["stage1_model"]["init_from"] = str(ckpt_path)
dest_models = build_models(_model_config(cfg2))
dest_critics = build_critics(_model_config(cfg2))
dest_trainers = build_stage_trainers(cfg2, dest_models, dest_critics, torch.device("cpu"), 4)
loaded = init_stages_from_checkpoints(dest_trainers)
assert len(loaded) == 1 and "stage1" in loaded[0]
dest_stage1_ema = dest_trainers["stage1"].ema_model
assert dest_stage1_ema is not None
_assert_state_dicts_equal(
source_trainers["stage1"].model.state_dict(), dest_trainers["stage1"].model.state_dict(), "init_from raw"
)
_assert_state_dicts_equal(
source_stage1_ema.state_dict(),
dest_stage1_ema.state_dict(),
"init_from ema",
)
# stage2 has no init_from set -- untouched fresh init, not the source's.
stage2_matches_source = all(
torch.equal(v, dest_trainers["stage2"].model.state_dict()[k])
for k, v in source_trainers["stage2"].model.state_dict().items()
)
assert not stage2_matches_source
def test_run_train_job_stage1_init_from_freeze_produces_rollout_capable_checkpoint(tmp_path):
"""The exact scenario gitea #42 exists for: retrain stage 2 alone against
a fixed, known-good stage 1, and still get a checkpoint giant rollout can
load (checkpoint_io.load_for_inference with require_stage2=True)."""
normalizer_dict = {
"cond": Normalizer().fit(np.zeros((1, COND_DIM), dtype=np.float32)).to_dict(),
"target": Normalizer().fit(np.zeros((1, X_DIM), dtype=np.float32)).to_dict(),
"sec_phys": Normalizer().fit(np.zeros((1, 2), dtype=np.float32)).to_dict(),
}
cfg = _base_cfg()
source_out = tmp_path / "source"
_run_train(cfg, source_out, normalizer_dict=normalizer_dict)
source_ckpt = torch.load(source_out / "best.pt", weights_only=False)
cfg2 = copy.deepcopy(cfg)
cfg2["stage1_model"]["init_from"] = str(source_out / "best.pt")
cfg2["stage1_model"]["freeze"] = True
retrain_out = tmp_path / "retrain"
_run_train(cfg2, retrain_out, normalizer_dict=normalizer_dict)
ctx = load_for_inference(retrain_out / "best.pt", torch.device("cpu"), "rollout", require_stage2=True)
assert ctx.stage1 is not None and ctx.stage2 is not None
retrain_ckpt = torch.load(retrain_out / "best.pt", weights_only=False)
for key, value in source_ckpt["model"].items():
assert torch.equal(value, retrain_ckpt["model"][key]), f"frozen stage1 {key} drifted across the retrain"
def test_stage_spec_from_config_omitted_decoder_and_particle_type_match_default_config(): def test_stage_spec_from_config_omitted_decoder_and_particle_type_match_default_config():
"""Regression for issues.md Issue 1: StageSpec.from_config's own fallback """Regression for issues.md Issue 1: StageSpec.from_config's own fallback
defaults for stage2_model.decoder/particle_type must equal defaults for stage2_model.decoder/particle_type must equal
@@ -831,3 +1146,127 @@ def test_wgan_physical_omits_grad_norm_slice_columns():
header = (out_dir / "metrics.csv").read_text().splitlines()[0].split(",") header = (out_dir / "metrics.csv").read_text().splitlines()[0].split(",")
assert "stage2/train/grad_norm_type_slice" not in header assert "stage2/train/grad_norm_type_slice" not in header
assert "stage2/train/grad_norm_cont_slice" not in header assert "stage2/train/grad_norm_cont_slice" not in header
# --- stage2_model.stage1_context = "sampled" (gitea #41) --------------------
def _sampled_ctx_cfg(ema_decay=0.999):
cfg = _base_cfg()
cfg["stage1_model"]["generator"] = "flow"
cfg["stage2_model"]["generator"] = "flow"
cfg["stage2_model"]["stage1_context"] = "sampled"
cfg["stage2_model"]["ctx_p_start"] = 0.0
cfg["stage2_model"]["ctx_p_end"] = 0.0
cfg["train"]["ema_decay"] = ema_decay
return cfg
def _build_sampled_trainers(cfg):
model_config = _model_config(cfg)
models = build_models(model_config)
critics = build_critics(model_config)
return build_stage_trainers(cfg, models, critics, torch.device("cpu"), total_train_batches=4)
def test_build_stage_trainers_attaches_stage1_only_under_sampled():
trainers = _build_sampled_trainers(_sampled_ctx_cfg())
assert trainers["stage2"].stage1_source is trainers["stage1"]
assert trainers["stage1"].stage1_source is None
def test_build_stage_trainers_leaves_stage1_source_none_under_truth():
"""Regression guard for the old silent no-op: 'truth' (the default) must
never attach a stage1_source, so _stage1_context short-circuits without
ever calling sample_stage1."""
cfg = _base_cfg()
trainers = _build_sampled_trainers(cfg)
assert trainers["stage2"].stage1_source is None
def test_stage1_context_sampled_calls_sample_stage1_and_differs_from_truth():
cfg = _sampled_ctx_cfg()
trainers = _build_sampled_trainers(cfg)
stage1, stage2 = trainers["stage1"], trainers["stage2"]
batch = _fake_batches(1, 4)[0]
cond_cont, cond_cat, x1_s1 = batch.cond_cont, batch.cond_cat, batch.target_s1
with patch("giant.training.trainers.sample_stage1", wraps=trainers_sample_stage1) as spy:
ctx = stage2._stage1_context(x1_s1, cond_cont, cond_cat, epoch=0)
assert spy.call_count == 1
assert spy.call_args.args[0] is stage1.sampling_model()
assert not torch.equal(ctx, x1_s1)
def test_stage1_context_truth_default_never_calls_sample_stage1():
cfg = _base_cfg()
trainers = _build_sampled_trainers(cfg)
stage2 = trainers["stage2"]
batch = _fake_batches(1, 4)[0]
cond_cont, cond_cat, x1_s1 = batch.cond_cont, batch.cond_cat, batch.target_s1
with patch("giant.training.trainers.sample_stage1", wraps=trainers_sample_stage1) as spy:
ctx = stage2._stage1_context(x1_s1, cond_cont, cond_cat, epoch=0)
assert spy.call_count == 0
assert torch.equal(ctx, x1_s1)
def test_stage1_context_val_epoch_none_uses_ground_truth_even_under_sampled():
cfg = _sampled_ctx_cfg()
trainers = _build_sampled_trainers(cfg)
stage2 = trainers["stage2"]
batch = _fake_batches(1, 4)[0]
cond_cont, cond_cat, x1_s1 = batch.cond_cont, batch.cond_cat, batch.target_s1
with patch("giant.training.trainers.sample_stage1", wraps=trainers_sample_stage1) as spy:
ctx = stage2._stage1_context(x1_s1, cond_cont, cond_cat, epoch=None)
assert spy.call_count == 0
assert torch.equal(ctx, x1_s1)
def test_stage1_context_sampled_preserves_stage1_training_mode():
"""Every sampler in giant/sample.py flips its model to .eval() as a side
effect with no restore of its own (see sample_flow). Sampling from the
RAW stage-1 model (ema_decay=0, so sampling_model() returns self.model,
the same weights the stage-1 trainer is actively training on) must not
silently leave it in eval mode for the rest of the epoch's stage-1
updates."""
cfg = _sampled_ctx_cfg(ema_decay=0.0)
trainers = _build_sampled_trainers(cfg)
stage1, stage2 = trainers["stage1"], trainers["stage2"]
stage1.train_mode()
assert stage1.model.training
batch = _fake_batches(1, 4)[0]
cond_cont, cond_cat, x1_s1 = batch.cond_cont, batch.cond_cat, batch.target_s1
stage2._stage1_context(x1_s1, cond_cont, cond_cat, epoch=0)
assert stage1.model.training
@pytest.mark.parametrize("stage2_generator", ["wgan", "flow"])
def test_build_stage_trainers_sampled_step_runs(stage2_generator):
"""Both trainer subclasses' call sites (FlowDDPMStageTrainer._compute,
WGANStageTrainer.step) must run end to end under 'sampled' and produce a
finite loss."""
cfg = _sampled_ctx_cfg()
cfg["stage2_model"]["generator"] = stage2_generator
trainers = _build_sampled_trainers(cfg)
trainer = trainers["stage2"]
batch = _fake_batches(1, 4)[0]
stats = trainer.step(batch, torch.device("cpu"), global_step=1)
loss_key = "g_loss" if stage2_generator == "wgan" else "loss"
assert math.isfinite(stats[loss_key])
def test_train_end_to_end_stage1_context_sampled():
"""Full train() run with stage1_context='sampled' must complete and
write a checkpoint + metrics.csv with finite losses throughout."""
cfg = _sampled_ctx_cfg()
with tempfile.TemporaryDirectory() as tmp:
out_dir = Path(tmp) / "run"
_run_train(cfg, out_dir)
assert (out_dir / "last.pt").exists()
with open(out_dir / "metrics.csv", newline="") as f:
rows = list(csv.DictReader(f))
assert len(rows) == cfg["train"]["epochs"]
assert all(math.isfinite(float(r["stage2/train/loss"])) for r in rows)
Generated
+200 -7
View File
@@ -5,19 +5,19 @@ resolution-markers = [
"python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'", "python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'", "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'", "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform == 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version >= '3.14' and sys_platform == 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version >= '3.14' and sys_platform == 'emscripten' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version < '3.14' and sys_platform == 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'emscripten' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version < '3.14' and sys_platform == 'emscripten' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"(python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and platform_machine != 'aarch64' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and platform_python_implementation != 'CPython' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda')", "(python_full_version == '3.13.*' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and platform_machine != 'aarch64' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and platform_python_implementation != 'CPython' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and sys_platform != 'darwin' and sys_platform != 'emscripten' and sys_platform != 'linux' and sys_platform != 'win32' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda')",
"(python_full_version < '3.13' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and sys_platform == 'darwin' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda')", "(python_full_version < '3.13' and platform_machine == 'aarch64' and platform_python_implementation == 'CPython' and sys_platform == 'linux' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda') or (python_full_version < '3.14' and sys_platform == 'darwin' and extra == 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda')",
"python_full_version >= '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version >= '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version < '3.14' and sys_platform == 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version < '3.14' and sys_platform == 'emscripten' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
"python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'", "python_full_version < '3.14' and sys_platform != 'emscripten' and sys_platform != 'win32' and extra != 'extra-5-giant-cpu' and extra != 'extra-5-giant-cuda'",
@@ -45,6 +45,19 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/99/91/8acff4f5e50511b911bbccb72b8628a49c68ce14148cd9f6431094859a90/annotated_types-0.8.0-py3-none-any.whl", hash = "sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0", size = 13427, upload-time = "2026-07-23T20:16:12.938Z" }, { url = "https://files.pythonhosted.org/packages/99/91/8acff4f5e50511b911bbccb72b8628a49c68ce14148cd9f6431094859a90/annotated_types-0.8.0-py3-none-any.whl", hash = "sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0", size = 13427, upload-time = "2026-07-23T20:16:12.938Z" },
] ]
[[package]]
name = "anyio"
version = "4.14.2"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "idna", marker = "sys_platform != 'emscripten' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
{ name = "typing-extensions", marker = "(python_full_version < '3.13' and sys_platform != 'emscripten') or (python_full_version >= '3.13' and extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda') or (sys_platform == 'emscripten' and extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
]
sdist = { url = "https://files.pythonhosted.org/packages/61/cc/a381afa6efea9f496eff839d4a6a1aed3bfafc7b3ab4b0d1b243a12573dd/anyio-4.14.2.tar.gz", hash = "sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f", size = 260176, upload-time = "2026-07-12T20:29:07.082Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/da/35/f2287558c17e29fafc8ef3daf819bb9834061cfa43bff8014f7df7f63bdc/anyio-4.14.2-py3-none-any.whl", hash = "sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494", size = 125813, upload-time = "2026-07-12T20:29:05.763Z" },
]
[[package]] [[package]]
name = "appnope" name = "appnope"
version = "0.1.4" version = "0.1.4"
@@ -134,6 +147,35 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/44/a1/70ebfffd6c6edc6034a547838ee46287c65ed89f710592ddc39c76b4a5a8/awkward_cpp-53-cp314-cp314t-win_arm64.whl", hash = "sha256:1be0c1d87d9f4fdf94b767a061df849f1bb21579d302b2996fb101527fc80a97", size = 551257, upload-time = "2026-06-08T12:31:56.319Z" }, { url = "https://files.pythonhosted.org/packages/44/a1/70ebfffd6c6edc6034a547838ee46287c65ed89f710592ddc39c76b4a5a8/awkward_cpp-53-cp314-cp314t-win_arm64.whl", hash = "sha256:1be0c1d87d9f4fdf94b767a061df849f1bb21579d302b2996fb101527fc80a97", size = 551257, upload-time = "2026-06-08T12:31:56.319Z" },
] ]
[[package]]
name = "bracex"
version = "3.0.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/ac/01/5f394b8bcd6e5b92f73130990960423bbb19711f906bd9fe9ea5557c667c/bracex-3.0.1.tar.gz", hash = "sha256:4e38e32392e4a4780fe15d644bfc7c8514057cfc3861e060b11814ce829c25e4", size = 44019, upload-time = "2026-07-20T13:43:00.335Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b8/8f/6f7273a7adb8d73fc8d21ede4376a3e475e52f98435c6007f69100dec8ca/bracex-3.0.1-py3-none-any.whl", hash = "sha256:6523ad83aeb5098a4ee597cff0f964442ff74e460bd3fafaffab6a013ff2288c", size = 11940, upload-time = "2026-07-20T13:42:59.268Z" },
]
[[package]]
name = "bump-my-version"
version = "1.5.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "httpx2" },
{ name = "pydantic" },
{ name = "pydantic-settings" },
{ name = "questionary" },
{ name = "rich" },
{ name = "rich-click" },
{ name = "tomlkit" },
{ name = "wcmatch" },
]
sdist = { url = "https://files.pythonhosted.org/packages/23/09/5b09ac74962eca809cbf7010a08ea6ad405852bdd53489209a9f473d775c/bump_my_version-1.5.1.tar.gz", hash = "sha256:5079e443ab8c9a9903f140b427ff9f6fe8dd54013a55a4cf48b89326f3a71c07", size = 1132060, upload-time = "2026-08-06T14:26:38.9Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/b3/0b/5885530f79d4400368b9d4dcb9b39274c0d52e633f7871e7fc6feceea1e3/bump_my_version-1.5.1-py3-none-any.whl", hash = "sha256:df3e2989d0d7fe704718feb24a5880f089b6b6369e427a4445b89c3adebfcff1", size = 65090, upload-time = "2026-08-06T14:26:37.083Z" },
]
[[package]] [[package]]
name = "certifi" name = "certifi"
version = "2026.7.22" version = "2026.7.22"
@@ -633,7 +675,7 @@ wheels = [
[[package]] [[package]]
name = "giant" name = "giant"
version = "0.3.2" version = "0.3.5"
source = { editable = "." } source = { editable = "." }
dependencies = [ dependencies = [
{ name = "numpy" }, { name = "numpy" },
@@ -666,6 +708,8 @@ cuda = [
] ]
dev = [ dev = [
{ name = "awkward" }, { name = "awkward" },
{ name = "bump-my-version" },
{ name = "git-cliff" },
{ name = "ipykernel" }, { name = "ipykernel" },
{ name = "matplotlib" }, { name = "matplotlib" },
{ name = "plotstyle" }, { name = "plotstyle" },
@@ -688,7 +732,9 @@ wandb = [
[package.metadata] [package.metadata]
requires-dist = [ requires-dist = [
{ name = "awkward", marker = "extra == 'convert'", specifier = ">=2.6,<3" }, { name = "awkward", marker = "extra == 'convert'", specifier = ">=2.6,<3" },
{ name = "bump-my-version", marker = "extra == 'dev'", specifier = ">=1.2,<2" },
{ name = "giant", extras = ["convert", "analysis", "geometry", "wandb"], marker = "extra == 'dev'" }, { name = "giant", extras = ["convert", "analysis", "geometry", "wandb"], marker = "extra == 'dev'" },
{ name = "git-cliff", marker = "extra == 'dev'", specifier = ">=2,<3" },
{ name = "ipykernel", marker = "extra == 'analysis'", specifier = ">=7.3.0" }, { name = "ipykernel", marker = "extra == 'analysis'", specifier = ">=7.3.0" },
{ name = "matplotlib", marker = "extra == 'analysis'", specifier = ">=3.8,<4" }, { name = "matplotlib", marker = "extra == 'analysis'", specifier = ">=3.8,<4" },
{ name = "numpy", specifier = ">=1.26,<3" }, { name = "numpy", specifier = ">=1.26,<3" },
@@ -713,6 +759,35 @@ requires-dist = [
] ]
provides-extras = ["cpu", "cuda", "dev", "geometry", "wandb", "convert", "analysis"] provides-extras = ["cpu", "cuda", "dev", "geometry", "wandb", "convert", "analysis"]
[[package]]
name = "git-cliff"
version = "2.13.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/62/57/b12494e2cbc3c9154c942e64659b5aec2b1ce9f12d07f6dc6167e2c63ae5/git_cliff-2.13.1.tar.gz", hash = "sha256:e949ea9c3951ba6037b99eec465162be2584f27f0836ace45f44d6f45650f8c6", size = 113119, upload-time = "2026-04-26T10:33:42.331Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/8f/dd/24768c3c0030710d36706c17b997d06aee27cb76b27ab2abb058ae254175/git_cliff-2.13.1-py3-none-macosx_10_12_x86_64.whl", hash = "sha256:08a9cb0ec760e165210ed22fefa295b6549a3520b420db995ccbb3620cbb1fbe", size = 7260035, upload-time = "2026-04-26T10:33:16.269Z" },
{ url = "https://files.pythonhosted.org/packages/22/df/842973ead79d27a58cd1eccd167191c0a71513e5c1e9dc30337dafdb7d36/git_cliff-2.13.1-py3-none-macosx_11_0_arm64.whl", hash = "sha256:e92cf470ecbe73f7d2963dfa80e8961a6b76888d0c19949b0f028a28a0a0470c", size = 6854384, upload-time = "2026-04-26T10:33:18.808Z" },
{ url = "https://files.pythonhosted.org/packages/7c/4d/6d6efa7d61be8632563990ccd402e401695e4a85a0bb1002f28730d03268/git_cliff-2.13.1-py3-none-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2e8d8e420adf6a36b97e0fbdbf2b07e47712199a9baf3675c9a759430243ea26", size = 7308164, upload-time = "2026-04-26T10:33:20.869Z" },
{ url = "https://files.pythonhosted.org/packages/f0/4a/98b8d2f53a2d0b7d313e98ab363ad7e0d6a514e878a8d678f22402cb0ec7/git_cliff-2.13.1-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1ab059d671565189faa4f3858b2fb42535aa39fe265ad95d84d5c116a2724fc7", size = 7687163, upload-time = "2026-04-26T10:33:23.096Z" },
{ url = "https://files.pythonhosted.org/packages/c7/07/cdd149b3909644aa3f0be7960406d9bbb38598f8618e039ac48cbb43ded6/git_cliff-2.13.1-py3-none-manylinux_2_28_aarch64.whl", hash = "sha256:a93db30da45967c42df607fbbc2092111fcd576b0ca9e2fbddd3f653d8c71be7", size = 7317670, upload-time = "2026-04-26T10:33:25.323Z" },
{ url = "https://files.pythonhosted.org/packages/2e/11/6d377a7f3113f6e26d87a28a32013eb05bd62bce176d6f8ee808e4868c1f/git_cliff-2.13.1-py3-none-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:df9f5a2bd16e5225030c9c2362e6ac70b34a906c0fbb83f8cbf5ae46932ba0d2", size = 7502294, upload-time = "2026-04-26T10:33:27.315Z" },
{ url = "https://files.pythonhosted.org/packages/80/6b/e1da9acf3aec99e6600be02b6ce0c9e8bd42d072e3120384514c905231e2/git_cliff-2.13.1-py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:c12276784d280aa6a7148d3e52ff139e891f4c97720cd9be581b19892ea39fe0", size = 7927258, upload-time = "2026-04-26T10:33:29.554Z" },
{ url = "https://files.pythonhosted.org/packages/ed/ea/9f2188a5e474e5f02193d9c1cdf7028773d483a3f508a1df4f1d93c9cc80/git_cliff-2.13.1-py3-none-musllinux_1_2_aarch64.whl", hash = "sha256:17da93dc605cbc48c762770402067fa726437cedbd62514f9caef6e0ccb58a43", size = 7308153, upload-time = "2026-04-26T10:33:31.589Z" },
{ url = "https://files.pythonhosted.org/packages/b3/70/5e2b2a0e42c07956f911e1eccd6dc6d79b96fc6c8a604a526c1ee0474c84/git_cliff-2.13.1-py3-none-musllinux_1_2_i686.whl", hash = "sha256:cd4a08cf3f638ec71d2ed451aa8673bef99e1107a36366153db97ca18d981655", size = 7502287, upload-time = "2026-04-26T10:33:33.894Z" },
{ url = "https://files.pythonhosted.org/packages/05/50/cc4c1d3d360621c0235d66d2c74472d9993d8aadf23ffa311eaf29d7a3aa/git_cliff-2.13.1-py3-none-musllinux_1_2_x86_64.whl", hash = "sha256:c5bb87f6e1db18be09e50c9d59234dc949713e20c2700e94eca378d22ad79719", size = 7927253, upload-time = "2026-04-26T10:33:36.205Z" },
{ url = "https://files.pythonhosted.org/packages/86/5d/717d30f37dad65a6cc5220b04a3ef1bc44c31fb56d0ce2895114e159df4f/git_cliff-2.13.1-py3-none-win32.whl", hash = "sha256:c8878972e0a6c26d9137fc406a611116239333578d95ac05064d2807920bd83c", size = 6718261, upload-time = "2026-04-26T10:33:38.1Z" },
{ url = "https://files.pythonhosted.org/packages/99/b2/99fac50978b9a90bfec0f1b89354a667ec83f4990301f6c708abce05484e/git_cliff-2.13.1-py3-none-win_amd64.whl", hash = "sha256:856d831a0bede9c258229dbd4d4c2b1c0810d8fce3d3882729669e8dc09c72bf", size = 7714969, upload-time = "2026-04-26T10:33:40.163Z" },
]
[[package]]
name = "h11"
version = "0.16.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" },
]
[[package]] [[package]]
name = "hepunits" name = "hepunits"
version = "2.4.6" version = "2.4.6"
@@ -722,6 +797,45 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/85/10/7f9c58d1ec6a0b7f7783fe552f3593f39cda30c2e1d7a9d148ae711e748d/hepunits-2.4.6-py3-none-any.whl", hash = "sha256:089c52c3b84ef67a159b5e9ee9bdd50e1a442e3fd0c101303cc409c1e9011c4d", size = 17090, upload-time = "2026-06-16T09:23:35.35Z" }, { url = "https://files.pythonhosted.org/packages/85/10/7f9c58d1ec6a0b7f7783fe552f3593f39cda30c2e1d7a9d148ae711e748d/hepunits-2.4.6-py3-none-any.whl", hash = "sha256:089c52c3b84ef67a159b5e9ee9bdd50e1a442e3fd0c101303cc409c1e9011c4d", size = 17090, upload-time = "2026-06-16T09:23:35.35Z" },
] ]
[[package]]
name = "httpcore2"
version = "2.11.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "h11", marker = "sys_platform != 'emscripten' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
{ name = "truststore", marker = "sys_platform != 'emscripten' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
]
sdist = { url = "https://files.pythonhosted.org/packages/30/7d/ee6787efd5fe675d7cfd5eb149e40ccb5bdfc7e7c9252edcf7825c38986f/httpcore2-2.11.0.tar.gz", hash = "sha256:82e6fc95d784e6ee22ebd4b2cb57df53a2efb13ad6a11260a236ecebbc5f50c7", size = 67532, upload-time = "2026-08-18T08:03:53.008Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/f3/54/e84a5c82ac0959d5e55b3970d326fd95306446b2bf0702302888745f7e5c/httpcore2-2.11.0-py3-none-any.whl", hash = "sha256:c7c899fbc6b8abb6e747dda427aa6f52934c45e191eabb986965164ebb02a908", size = 83061, upload-time = "2026-08-18T08:03:50.894Z" },
]
[[package]]
name = "httpx2"
version = "2.11.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "anyio", marker = "sys_platform != 'emscripten' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
{ name = "httpcore2", marker = "sys_platform != 'emscripten' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
{ name = "httpx2-jsfetch", marker = "sys_platform == 'emscripten' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
{ name = "idna" },
{ name = "truststore", marker = "sys_platform != 'emscripten' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
{ name = "typing-extensions", marker = "python_full_version < '3.13' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
]
sdist = { url = "https://files.pythonhosted.org/packages/d5/4d/b3fcae38f29bfb0f300517d085c488f41f65e5b0a73023976b2122f568cc/httpx2-2.11.0.tar.gz", hash = "sha256:ea01b2e8febfb026e2601814c77ecb1e64fff114a87bc789cb520e67f27e7809", size = 99617, upload-time = "2026-08-18T08:03:53.691Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/39/29/f60bcf54028601920c0ce3da81537bcd0af4ee19dd9672af85dc9dfd60e0/httpx2-2.11.0-py3-none-any.whl", hash = "sha256:c9790f62a327110f52a099f1e2030cbe32f78b28781ad68fe58bfd6f23e73ab0", size = 95043, upload-time = "2026-08-18T08:03:52.004Z" },
]
[[package]]
name = "httpx2-jsfetch"
version = "1.0"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/cd/c4/0e5636363151a2a1795e0a77617168b9ca438e1748ec05fc9b5687f93d64/httpx2_jsfetch-1.0.tar.gz", hash = "sha256:70a0e3eabfef7cce5ad9c629f7d01ca05e418f586646f4ddf14782e4c1454c60", size = 6872, upload-time = "2026-08-07T00:13:07.492Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/9b/43/832f631d32e4f1211caa2ba368317739fe71f0b8530e4c9d15dc454bac2a/httpx2_jsfetch-1.0-py3-none-any.whl", hash = "sha256:cb916b707601e69a07721aabc8f3f6659be3a6893bc1ff5c6f9e02241df2da32", size = 6382, upload-time = "2026-08-07T00:13:06.567Z" },
]
[[package]] [[package]]
name = "idna" name = "idna"
version = "3.18" version = "3.18"
@@ -1780,6 +1894,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/fa/c3/7c8b240552251faf6b3a957db200fcfbbcec36763c050428b601e0c9b83b/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0", size = 2147590, upload-time = "2026-05-06T13:39:29.883Z" }, { url = "https://files.pythonhosted.org/packages/fa/c3/7c8b240552251faf6b3a957db200fcfbbcec36763c050428b601e0c9b83b/pydantic_core-2.46.4-graalpy312-graalpy250_312_native-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0", size = 2147590, upload-time = "2026-05-06T13:39:29.883Z" },
] ]
[[package]]
name = "pydantic-settings"
version = "2.15.0"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "pydantic" },
{ name = "python-dotenv" },
{ name = "typing-inspection" },
]
sdist = { url = "https://files.pythonhosted.org/packages/68/ca/31c57507b13119d7d3cfa1576dad2911a4861e3be07b579395f4e9d393f9/pydantic_settings-2.15.0.tar.gz", hash = "sha256:694b793e84f766ba76a90ebdefc01d0a9a045dab0382bee70393da93712ad117", size = 261253, upload-time = "2026-08-07T09:24:57.419Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/30/a4/2bffa9f8e804325a09867f0e9d30795c80ea9f8d62560bd1b6ad6220eb2f/pydantic_settings-2.15.0-py3-none-any.whl", hash = "sha256:0ba092c291c94baceb5eff768aa0d56400a457585bc0175925a5a5510303da42", size = 69413, upload-time = "2026-08-07T09:24:55.839Z" },
]
[[package]] [[package]]
name = "pygments" name = "pygments"
version = "2.20.0" version = "2.20.0"
@@ -1840,6 +1968,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" }, { url = "https://files.pythonhosted.org/packages/ec/57/56b9bcc3c9c6a792fcbaf139543cee77261f3651ca9da0c93f5c1221264b/python_dateutil-2.9.0.post0-py2.py3-none-any.whl", hash = "sha256:a8b2bc7bffae282281c8140a97d3aa9c14da0b136dfe83f850eea9a5f7470427", size = 229892, upload-time = "2024-03-01T18:36:18.57Z" },
] ]
[[package]]
name = "python-dotenv"
version = "1.2.3"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/6a/53/ed9d74092561d4b01a2ef1349d52cdbc135e526c245f366b089cfca6de49/python_dotenv-1.2.3.tar.gz", hash = "sha256:a20a594dabeaa385725aa239d5244871c143ecb356add8a20fcf23773a6c3a35", size = 58945, upload-time = "2026-08-16T16:54:54.067Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/0d/17/c5c6b53ddc18f297992099b3d9ec16c855c0ccc83263a21fe4d1c625ec6c/python_dotenv-1.2.3-py3-none-any.whl", hash = "sha256:904552145e8bfed22162c09dab1c2b9b54fefa7b23ba780f4f26ca0316b0f0d9", size = 22780, upload-time = "2026-08-16T16:54:52.473Z" },
]
[[package]] [[package]]
name = "pyyaml" name = "pyyaml"
version = "6.0.3" version = "6.0.3"
@@ -1929,6 +2066,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/81/d6/4bfbb40c9a0b42fc53c7cf442f6385db70b40f74a783130c5d0a5aa62228/pyzmq-27.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dc5dbf68a7857b59473f7df42650c621d7e8923fb03fa74a526890f4d33cc4d7", size = 575170, upload-time = "2025-09-08T23:09:01.418Z" }, { url = "https://files.pythonhosted.org/packages/81/d6/4bfbb40c9a0b42fc53c7cf442f6385db70b40f74a783130c5d0a5aa62228/pyzmq-27.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:dc5dbf68a7857b59473f7df42650c621d7e8923fb03fa74a526890f4d33cc4d7", size = 575170, upload-time = "2025-09-08T23:09:01.418Z" },
] ]
[[package]]
name = "questionary"
version = "2.1.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "prompt-toolkit" },
]
sdist = { url = "https://files.pythonhosted.org/packages/f6/45/eafb0bba0f9988f6a2520f9ca2df2c82ddfa8d67c95d6625452e97b204a5/questionary-2.1.1.tar.gz", hash = "sha256:3d7e980292bb0107abaa79c68dd3eee3c561b83a0f89ae482860b181c8bd412d", size = 25845, upload-time = "2025-08-28T19:00:20.851Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/3c/26/1062c7ec1b053db9e499b4d2d5bc231743201b74051c973dadeac80a8f43/questionary-2.1.1-py3-none-any.whl", hash = "sha256:a51af13f345f1cdea62347589fbb6df3b290306ab8930713bfae4d475a7d4a59", size = 36753, upload-time = "2025-08-28T19:00:19.56Z" },
]
[[package]] [[package]]
name = "requests" name = "requests"
version = "2.34.2" version = "2.34.2"
@@ -1957,6 +2106,20 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" }, { url = "https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl", hash = "sha256:33bd4ef74232fb73fe9279a257718407f169c09b78a87ad3d296f548e27de0bb", size = 310654, upload-time = "2026-04-12T08:24:02.83Z" },
] ]
[[package]]
name = "rich-click"
version = "1.9.8"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "click" },
{ name = "colorama", marker = "sys_platform == 'win32' or (extra == 'extra-5-giant-cpu' and extra == 'extra-5-giant-cuda')" },
{ name = "rich" },
]
sdist = { url = "https://files.pythonhosted.org/packages/f7/ea/21e4867ea0ef881ffd4c0550fc21a061435e50d6324bcd034396633cbc18/rich_click-1.9.8.tar.gz", hash = "sha256:4008f921da88b5d91646c134ec881c1500e5a6b3f093e90e8f29400e09608371", size = 75363, upload-time = "2026-05-28T19:54:59.144Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/6d/97/a87901aef6b7e7e4a34c6dd6cc17dca8594a592ef9d9dd765fca2b7facf7/rich_click-1.9.8-py3-none-any.whl", hash = "sha256:12873865396e6927835d4eabb1cc3996edcd65b7ac9b2391a29eca4f335a2f93", size = 72189, upload-time = "2026-05-28T19:54:57.867Z" },
]
[[package]] [[package]]
name = "ruff" name = "ruff"
version = "0.15.17" version = "0.15.17"
@@ -2149,6 +2312,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" }, { url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" },
] ]
[[package]]
name = "tomlkit"
version = "0.15.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/94/96/e07752635b98536177fa1f37671c8f3cdde2e724c6bcf6034b2cfb571565/tomlkit-0.15.1.tar.gz", hash = "sha256:e25bbf38843005246210a12982776f27f99cb9be67160e14434d0c0d21ee1e97", size = 180129, upload-time = "2026-07-17T01:48:04.562Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/13/bc/8c13eb66537dce1d2bd3a57132902f38d0e7f5bb46fa9f4daed9fe9d76ee/tomlkit-0.15.1-py3-none-any.whl", hash = "sha256:177a05aece5a8ca5266fd3c448abb47b8d352f09d477d3ca8332db4d89b24304", size = 49449, upload-time = "2026-07-17T01:48:05.728Z" },
]
[[package]] [[package]]
name = "torch" name = "torch"
version = "2.3.1" version = "2.3.1"
@@ -2270,6 +2442,15 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/96/8d/1080ee4c231f361b6ce4470d556c8c435b67c7e0753aaa641497ee92f88b/traitlets-5.15.1-py3-none-any.whl", hash = "sha256:770a53705f84b81ac107e83a1b3328ff2dae16094d8fc3cfc004e4b22dfd8e92", size = 85858, upload-time = "2026-06-03T12:26:04.395Z" }, { url = "https://files.pythonhosted.org/packages/96/8d/1080ee4c231f361b6ce4470d556c8c435b67c7e0753aaa641497ee92f88b/traitlets-5.15.1-py3-none-any.whl", hash = "sha256:770a53705f84b81ac107e83a1b3328ff2dae16094d8fc3cfc004e4b22dfd8e92", size = 85858, upload-time = "2026-06-03T12:26:04.395Z" },
] ]
[[package]]
name = "truststore"
version = "0.10.4"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/53/a3/1585216310e344e8102c22482f6060c7a6ea0322b63e026372e6dcefcfd6/truststore-0.10.4.tar.gz", hash = "sha256:9d91bd436463ad5e4ee4aba766628dd6cd7010cf3e2461756b3303710eebc301", size = 26169, upload-time = "2025-08-12T18:49:02.73Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/19/97/56608b2249fe206a67cd573bc93cd9896e1efb9e98bce9c163bcdc704b88/truststore-0.10.4-py3-none-any.whl", hash = "sha256:adaeaecf1cbb5f4de3b1959b42d41f6fab57b2b1666adb59e89cb0b53361d981", size = 18660, upload-time = "2025-08-12T18:49:01.46Z" },
]
[[package]] [[package]]
name = "ty" name = "ty"
version = "0.0.50" version = "0.0.50"
@@ -2394,6 +2575,18 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/07/78/75b6827a6665337a715c5347c5edbd84eca660f7a0f48d8d6d24d1f66bee/wandb-0.28.1-py3-none-win_arm64.whl", hash = "sha256:4aa07f13dd3bcac2c0524c8d0f49f76e83ab5c1054fd09f3b1a436cfcde146a6", size = 22299006, upload-time = "2026-07-16T18:47:02.71Z" }, { url = "https://files.pythonhosted.org/packages/07/78/75b6827a6665337a715c5347c5edbd84eca660f7a0f48d8d6d24d1f66bee/wandb-0.28.1-py3-none-win_arm64.whl", hash = "sha256:4aa07f13dd3bcac2c0524c8d0f49f76e83ab5c1054fd09f3b1a436cfcde146a6", size = 22299006, upload-time = "2026-07-16T18:47:02.71Z" },
] ]
[[package]]
name = "wcmatch"
version = "11.0.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "bracex" },
]
sdist = { url = "https://files.pythonhosted.org/packages/57/43/30e407989e313677dbb9d5f045f966549a7254834571e342eaa4b55cc67b/wcmatch-11.0.1.tar.gz", hash = "sha256:1ea2b4fa678b8ca268253798d5963935df39132d47c3e241c0a0732224005e7d", size = 144662, upload-time = "2026-08-14T15:20:40.477Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ce/77/7a02b0f05b3ffcdbef9719ce3ee0b508d6a29b58e95299f1580055671db3/wcmatch-11.0.1-py3-none-any.whl", hash = "sha256:fd149ecddb9f0a88ea780017d6dde17c994e494e7f7303d4e3c9d6251f978f4b", size = 43449, upload-time = "2026-08-14T15:20:39.379Z" },
]
[[package]] [[package]]
name = "wcwidth" name = "wcwidth"
version = "0.8.1" version = "0.8.1"