129 Commits

Author	SHA1	Message	Date
lars	78978769f6	Add bf16 autocast to the training loop (gitea #47) ... giant/ had no autocast/GradScaler/torch.compile anywhere despite the project's ~10x-native-Geant4 eval-budget target. This adds bf16 mixed precision to the training step (both FlowDDPMStageTrainer and WGANStageTrainer) via a new train.precision config key ("fp32" default, "bf16" opt-in) and giant.training.amp.resolve_autocast. torch.compile is a separate, much larger surface (data-dependent routed dispatch, the autoregressive sampler's per-token control flow, arbitrary rollout batch sizes) and is left for a follow-up issue, per discussion. Scope decisions made during planning: - fp32 + bf16 only, no fp16/GradScaler. fp16 breaks two things in this codebase: routers.py's three 1e-8 epsilons sit below fp16's ~6e-8 subnormal floor, and gradient_penalty's grad norm overflows fp16's range at ordinary early-WGAN-GP gradient magnitudes. Every training GPU in the fleet (A100/L40S/H200/RTX 4070) has native bf16; only pre-Ampere V100s would need fp16. - resolve_autocast raises loudly if bf16 is requested on hardware that can't do it, rather than silently falling back to fp32. - Autocast wraps the training step only; val_loss (and the best-checkpoint selection it drives) stays fp32 so it's comparable across every run recorded so far. - _route_forward's mixture accumulator (giant/model/trunks.py) was a hard-fp32 torch.zeros with no dtype, so under autocast a RoutedTrunk silently returned a different output dtype than an unrouted ExpertTrunk purely because router.enabled was set. Fixed to match the experts' own dtype; the gate weights (forced fp32 for their own numerical stability) are cast down before combining, so the mixture's numerics stay solid without reintroducing the dtype split. - Added explicit fp32 guards (autocast(enabled=False)) around spots that are correct in fp32 but degrade quietly rather than crash in bf16: the router's balance/entropy losses and gate softmax, the stage-2 stick-breaking cumprod, and gradient_penalty's double-backward + grad norm. Benchmarked on the local RTX 4070 against configs/baseline.toml's hyperparams (hidden_dim 512/6 blocks, bs 4096) on a synthetic dataset: bf16 gave 1.05-1.35x training throughput and 18-33% lower peak GPU memory across one-shot/routed/autoregressive stage-2 configs, with the autoregressive path (the dominant cost per baseline.toml) benefiting most on both axes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 15:36:12 +02:00
lars	87e37ebe14	Add per-stage init_from/freeze (gitea #42) ... stage{1,2}_model.active = false already trains one stage alone, but the checkpoint it writes holds only that stage, so giant rollout refuses it -- the "retrain stage 2 alone against a fixed, known-good stage 1" experiment the 2026-08-03 species failure calls for wasn't runnable end to end. Adds stage{1,2}_model.init_from (a checkpoint .pt to load this stage's weights from before training) and .freeze (never update them), symmetric across both stages. Both stages stay active = true, so both get built and both land in the output checkpoint -- the frozen stage is merely initialized from disk instead of from scratch. Decisions made during planning: - Soft freeze: forward/backward still run every batch (loss/grad_norm stay meaningful, no autograd special-casing), only optimizer.step() (and, for the frozen stage, lr_sched.step()/EMA update) is skipped -- weights are byte-identical for the whole run. This is StageTrainer._step_optimizer, shared by the non-adversarial path and both halves (generator + critic) of the WGAN path, so a frozen WGAN stage's critic freezes too. - validate_config requires init_from whenever freeze = true, unless the run is a --resume (a resumed frozen stage's weights come from the resume checkpoint instead) -- freezing a randomly-initialized model is almost certainly a mistake. - CLI flags on both `giant train` and `giant new-run` (--stage{1,2}-init-from/--stage{1,2}-freeze), matching every other per-stage model knob's existing treatment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 14:23:20 +02:00
lars	48faaee79d	Implement stage2_model.stage1_context = "sampled" (gitea #41) ... Stage 2 was trained on ground-truth stage-1 outcomes but deployed on sampled ones, and in a rollout that gap compounds over every step of every track — the same train/inference gap teacher_forcing="scheduled" already closes within stage 2, just never applied at the stage boundary. "sampled" was declared in the schema but rejected loudly by validate_config as unimplemented; this lands the real implementation. Mirrors the existing scheduled-sampling precedent rather than a hard switch: new stage2_model.ctx_p_start/ctx_p_end (defaults 1.0 -> 0.0) linearly ramp P(condition on ground truth) from epoch 0 to the final epoch, so stage 2 doesn't chase a wildly moving stage-1 target early in training. Per the plan discussed with the user: the sample is drawn from stage 1's sampling_model() (EMA weights when present, matching what inference actually deploys), mixed per example via a Bernoulli draw (never blended within a row), and validation always uses the ground truth regardless of the schedule. Fixes a latent bug the same pattern would otherwise have hit: every sampler in giant/sample.py flips its model to .eval() with no restore, so sampling from the raw (non-EMA) stage-1 model mid-step now explicitly restores its .training flag afterward to avoid silently corrupting stage 1's own training mode for the rest of the epoch. validate_config now enforces stage1_context in {"truth", "sampled"}, requires both stages active for "sampled" (nothing to sample from otherwise), range-checks ctx_p_start/ctx_p_end, and rejects the ctx_p_start = ctx_p_end = 1.0 configuration as an unadvertised no-op identical to "truth". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 12:27:13 +02:00
lars	cc9646f279	Add giant model summary command (gitea #46) ... giant model summary --config config.toml builds the resolved Stage1/Stage2 graph from a config with no dataset attached (pdg_vocab/mat_vocab are supplied as placeholders via --pdg-vocab/--mat-vocab, since the real training vocab is dataset-derived) and prints per-module parameter counts, trunk in/out widths, which heads exist, and which conditioning/stage1_model/stage2_model config keys actually shaped the build. The consumed-keys half uses differential probing rather than static identifier matching: build once for a fingerprint (submodule presence, every parameter's/buffer's shape+dtype, every plain scalar attribute a module stores on itself), then perturb one leaf at a time, rebuild, and compare. A changed fingerprint (or a raise) means the key is consumed; no change means it's inert *under this particular config* -- e.g. any stage1_model.router.* key when router.enabled=false. A curated _NOT_BUILD_TIME table separates keys legitimately owned by the trainer/sampler/rollout (loss weights, WGAN-GP hyperparameters, teacher-forcing schedules) from genuinely-inert ones, verified against those call sites. A few config keys branch on equality against one specific string literal (n_sec.owner=="stage1", n_sec.mode=="stop_token", particle_type.target=="physical"); a single generic sentinel probe missed all three since the config's current value and the sentinel landed in the same branch, so those three leaves get their real alternative value tried too (_STRING_ALTERNATIVES). giant.config.leaf_paths is promoted out of tests/test_config_consumed_keys.py (previously a private test-local duplicate) so both audits -- the static per-identifier one and this new runtime per-config one -- walk the exact same DEFAULT_CONFIG tree. ExpertTrunk/RoutedTrunk now also expose in_dim (out_dim already existed), needed to report trunk widths generically. Decisions made during planning: --pdg-vocab/--mat-vocab default to 300 and len(MATERIAL_PROPERTIES); the consumed-keys report is scoped to conditioning/stage1_model/stage2_model only (train/meta are out of scope for a model-only build); the module tree prints every submodule at any depth. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 11:48:02 +02:00
lars	c1c4957e2f	Implement n_sec.mode = "stop_token" for the AR secondary decoder (gitea #40) ... Stage 2's autoregressive decoder still predicted multiplicity the v0.2 way: a one-shot n_sec_head classifier over conditioning alone, run before any secondary token existed, with the AR loop then always executing k_max slots and discarding the tail. This adds a real per-slot EOS mechanism instead: - Stage2Autoregressive gains a stop_head (build_stop_head=True) that predicts P(n_sec == k | prefix) at each slot, mutually exclusive with n_sec_head (n_sec.mode = "stop_token" builds no n_sec_head at all). - sample_secondaries_ar accepts n_sec_pred=None to drive generation off the stop head instead of a pre-resolved count: each row stops the first slot its stop logit fires (stage2_model.n_sec.stop_sampling = "greedy" — the default, threshold at 0 — or "sample", a Bernoulli draw), and the whole batch loop breaks once every row has stopped, so cost scales with the realized n_sec instead of a fixed k_max. Passing n_sec_pred explicitly (the scheduled-sampling self-sample path) is unchanged. - resolve_n_sec returns None for a stop-token decoder instead of raising; rollout.py/cli.py/validate.py now derive the realized count from sample_stage2's returned sec_valid (sec_valid.sum(-1)) after sampling, rather than resolving it up front — a no-op reordering under every other n_sec.mode, where sec_valid was already built from n_sec_pred. - Training: _stop_target_and_mask (giant/training/stage2_inputs.py) builds the per-slot target/mask (one slot wider than the existing token-content sec_mask, since the stop slot itself needs supervision) and StageTrainer._stop_loss trains it with masked BCE, gated on stop_head exactly like _n_sec_loss gates on n_sec_head. Wired into both the flow/ddpm trainer and the WGAN trainer (whose skip_g_step now also checks stop_head), weighted by the existing stage2_model.n_sec.lambda — the stop head replaces n_sec_head under this mode, so no new weight key. - validate_config now accepts stop_token (requires decoder="autoregressive" and n_sec.owner="stage2") instead of always rejecting it. Decisions made during planning: stop_sampling defaults to "greedy" for deterministic rollouts; the stop head reuses stage2_model.heads.n_sec's HeadConfig shape and stage2_model.n_sec.lambda's weight rather than adding new config keys, since the two heads never coexist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 10:44:14 +02:00
lars	867a07da2b	Merge branch 'master' into fix/issue-61	2026-08-17 09:51:37 +02:00
lars	a746efb6e1	Clamp analysis histogram bins before the i32 cast, not after (gitea #61) ... _bin_expr in giant/analysis/reduce.py clipped the bin index to [0, nbins-1] only after casting it to Int32, so the clip never got a chance to run: a rollout step_length of 1.0725e10 mm against fixed edges [2.9e-5, 94.04] with 50 bins produces a raw index of ~5.7e9, which overflows i32 and fails the strict cast, killing the whole compute-one job. Same failure mode for +/-inf. Clamp in f64 first, then cast to Int32. NaN has no edge to clamp to, so it maps to null and is dropped in the two callers (hist1d, profile_partial) — matching what np.histogram does with NaN, and what profile_partial needs anyway since a null bin index would break its np.add.at. This reimplements commit 313373c, which fixed the same bug but landed on a branch (fix/rollout-negative-secondary-mass) that forked off a stale master and was never merged; reduce.py has since diverged enough that the original diff no longer applies cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 09:40:09 +02:00
lars	bacc8763d0	Clip raw predicted log_mass in decode_secondaries (gitea #54) ... decode_secondaries inverted a secondary's raw predicted log_mass with inv_log_transform (exp(y) - eps) unclipped. log_mass is a raw regression output, not itself the result of log_transform, so it isn't guaranteed to land in the range that round-trips cleanly: too negative and exp(y) undershoots eps, making the result go slightly negative; too positive and exp(y) overflows float32 to inf. Either one crashes the next rollout step, since a track descended from that secondary feeds its mass back in as conditioning, and log_transform raises on a non-finite input. Clip log_mass to [log(_EPS), _LOG_MASS_MAX] before inverting, guaranteeing a finite, non-negative mass. _LOG_MASS_MAX=80.0 matches the value from the stale fix/rollout-negative-secondary-mass branch (comfortably below float32's ~88.7 overflow point, far beyond any physical particle mass a converged model would predict) — that branch had already implemented this fix but forked before gitea #35/#36 and couldn't be merged as-is, so this reimplements it fresh against current master and leaves the stale branch untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 09:31:37 +02:00
lars	d25dfc0343	Let dwarf warm-cache take --config so it can't under-warm a config's cache keys (gitea #59) ... warm-cache built its config from DEFAULT_CONFIG with only a handful of flags overridable, so it had no way to express settings like stage2_model.particle_type.n_classes. configs/baseline.toml sets that to 32; warm-cache always warmed the pdg top-N map under the emb_dim default (16) instead, so a `giant train --config configs/baseline.toml` run silently missed the cache and repaid the full parquet scan warm-cache exists to avoid. warm-cache now accepts the same --config a training run takes and resolves every value run_setup_stage needs (val_fraction/seed, conditioning types, both stages' router, particle_type.n_classes, ...) from one gconfig.merge_cli_overrides + validate_config pass, exactly like giant train's own pipeline does — so warming and training are guaranteed to agree. Per user decision, --config is mutually exclusive with the individual --val-fraction/--seed/--particle-conditioning/ --material-conditioning/--router*/flags (rejected outright rather than silently layered on top), since a hardcoded CLI default clobbering an unset config value is the same failure mode one level down. Also drops a hardcoded stage2_model.router/k_max override that was a no-op against today's defaults but would have clobbered a config setting either one away from its default — same bug class. Adding validate_config surfaced that the existing test_warm_cache_router_process_warms_proc_map test was warming a router.type="process" + conditioning.particle.type="physical" (the CLI's old hardcoded default) combination that giant train's own validate_config would already reject as incompatible — fixed by passing --particle-conditioning embedding, which is what a working --router-type process run actually requires. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-17 08:54:14 +02:00
lars	c4b12b5e7a	Pass ConditioningAxisConfig/ParticleTypeConfig themselves instead of raw dicts (gitea #38) ... build_models/build_critics parsed model_config into frozen dataclasses (ConditioningConfig, Stage2ModelConfig, ...) but then threw the parsed sub-objects away and passed the original raw dicts (conditioning["particle"], s2_spec.particle_type.to_dict()) down into ConditionEncoder/StageModel/etc, which re-read them with their own hardcoded .get(key, default) fallbacks — each an independent copy of a fact the dataclass already stated once. Worst instance: giant/training/trainers.py:236 converted an already-parsed ParticleTypeConfig back into a dict for no reason. Threads ConditioningAxisConfig (particle_cfg/material_cfg) and ParticleTypeConfig (particle_type_cfg) as the actual dataclass instances through every signature that used to type them dict: ConditionEncoder, StageModel/CriticModel, resolve_type_n_classes/stage2_type_dim/ stage2_trunk_sec_dim, giant/model/builders.py, giant/sample.py, giant/training/stage2_inputs.py, giant/training/trainers.py (StageSpec/ StageTrainer), giant/pipeline.py, giant/rollout.py, giant/validate.py — so ty now catches a misspelled field instead of it silently falling back. No config-schema change: config.toml/checkpoint model_config keep the same nested-dict shape; only what happens after the existing X.from_dict(...) parse changes. User-confirmed scope decision: both axes (particle_cfg/material_cfg and particle_type_cfg), not just the more heavily-duplicated particle_type_cfg axis, and not stopping at the two most literal parse-then-discard round trips — matching the issue's own proposal. Preserved-default decision: StageModel's particle_type_cfg=None sentinel (hit only by direct/test construction — build_models always passes an explicit particle_type) still resolves to ParticleTypeConfig(target= "physical"), not ParticleTypeConfig()'s own target="onehot" config-file default — switching it would have silently grown an unused, gradient-less type_head on every test that constructs Stage2OneShot/Stage2Autoregressive without particle_type_cfg=, breaking their "every param has a grad" checks. New tests in tests/test_network.py: ConditionEncoder/StageModel store the exact ConditioningAxisConfig/ParticleTypeConfig instance passed in (identity, not just equality) — no internal dict round-trip — and build_models's output carries real dataclass instances end to end, not the plain dicts it produced before this fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 15:03:55 +02:00
lars	1a3c907571	Give Stage1Model/Stage2OneShot/Stage2Autoregressive a shared StageModel base (gitea #39) ... Stage1Model, Stage2OneShot and Stage2Autoregressive each independently implemented ~90 near-identical lines of __init__ scaffolding: build-or-share cond_enc, particle_type_cfg normalisation, objective -> time_emb -> merged_cond_dim -> build_trunk, and the n_sec_head/type_head classifier heads (plus their identical RuntimeError guards). Now unblocked by #33 (trunk registry), #34 (block-conditioning registry) and #36 (build_mlp_head), which settled what belongs in the shared base. Adds StageModel(nn.Module) owning all of that: __init__ builds/shares cond_enc and normalises particle_type_cfg; _build_trunk_and_heads, called by each subclass after it sets up its own conditioning-assembly modules (cond_enc alone for Stage1Model, a context-fusion path for the two Stage2 classes), builds the objective/time embedding/trunk and the n_sec_head/type_head guarded by the shared _require_n_sec_head/ _require_type_head (Stage1Model overrides the n_sec guard since its message points at stage 2, not stage 1). Public __init__ signatures, attribute names, and forward/predict_* behaviour are unchanged. Verified with a pre/post state_dict-key-set diff against the pre-refactor classes (bit-identical) before writing this commit, plus new parametrized tests pinning each class's state_dict key set and the generator -> time_emb contract the base now owns. tests/test_migration_ v02_v03.py's existing bit-identical old-vs-new forward comparison and the rest of tests/test_network.py's per-class coverage pass unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 14:37:58 +02:00
lars	4692cee699	Give the cond_cat/cond_cont column layout one owner (gitea #37) ... The conditioning arrays' column order was written down three times — twice in giant/data/transforms.py (build_cond_features and build_features each built cond_cont and cond_cat from scratch) and again in giant/model/encoders.py (cat_col_layout, plus hand-written COND_DIM_BASE + PARTICLE_PHYS_DIM slicing in ConditionEncoder). The three were held in sync only by parallel comments, so a wrong column order produced silently mis-indexed features rather than an exception. The drift had already happened, twice, both times in build_features: - 5b63dfd added per-axis vocab-lookup strictness (an out-of-vocab pdg/material must not KeyError under "physical"/"onehot", where the index is never read) to build_cond_features only. - _cond_normalizer_transform's legacy-normalizer padding, which keeps a pre-physical-conditioning 8-wide cond normalizer loadable, was likewise only wired into build_cond_features — so `giant predict` on such a checkpoint died with a broadcast error. New giant/cond_layout.py holds a frozen CondLayout built from the (particle, material) mode pair, exposing named cond_cont slices (base/particle_phys/material_phys) and cond_cat columns (PDG_COL/MAT_COL/particle_topn_col/material_topn_col/cat_dim). Both builders now share one _build_cond_arrays, ConditionEncoder reads its slices off the same object, and PdgRouter/ProcessRouter use the named dense-vocab columns instead of literal 0/1. CondLayout also absorbs the two duplicated axis-type validations, keeping their message text verbatim. Decisions taken while planning: - Scope is CondLayout only. The issue's second half — a CONDITIONING_AXIS_REGISTRY registering (feature_columns, encoder_module) as a pair — is deferred: it would force ConditioningConfig's fixed particle/material fields into a dynamic axis map and ripple through pipeline.py, checkpoint_io.py and rollout.py, i.e. a config-schema break with no consumer yet. - The two divergences above are unified onto build_cond_features' behaviour rather than preserved as parameters, so the new single source of truth doesn't carry the old split forward. Each gets a regression test that fails before this commit. - cat_col_layout is replaced outright (deleted, dropped from network.py's __all__, its four tests rewritten against CondLayout) rather than kept as a wrapper — two spellings of the same fact is the defect itself. cond_cat's width is now the layout's call rather than "did the caller pass a map", so an "onehot" axis without its top-N map raises instead of yielding a narrower array that ConditionEncoder would index out of bounds. pipeline.py's normalizer-fitting pass reads only cond_cont but had to be handed the maps to satisfy that. No parameter, buffer or state_dict change; existing checkpoints load unchanged, and the protected migration surfaces are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 14:16:20 +02:00
lars	593c5f4d34	Deduplicate n_sec_head/type_head MLPs into build_mlp_head (gitea #36) ... The same two-layer classifier head (Linear(cond_out_dim, hidden_dim // 2) -> SiLU -> Linear(hidden_dim // 2, out_dim)) was hand-rolled five times in giant/model/models.py: Stage1Model.n_sec_head, Stage2OneShot.n_sec_head/ .type_head, and Stage2Autoregressive.n_sec_head/.type_head. The `// 2` ratio and fixed 2-layer depth were undocumented magic numbers, and both n_sec accuracy and secondary-species accuracy are known weak spots that were untunable independently of the trunk they hang off. Adds `build_mlp_head(in_dim, out_dim, hidden, depth, act)` to giant/model/layers.py (depth=1 is a bare Linear; depth>=2 matches the old hardcoded shape exactly), and a new `HeadConfig` (hidden_ratio, depth) dataclass in giant/config.py, wired in as `stage1_model.heads.n_sec` and `stage2_model.heads.{n_sec,type}` — split per head type (not one shared block per stage) since n_sec and species prediction are called out as separate weak spots that may want independent capacity. Defaults (hidden_ratio=0.5, depth=2) reproduce the old hardcoded architecture bit-for-bit, so every existing config.toml and migrated v0.2 checkpoint is unaffected; no changes were needed to migrate_config or the legacy migration surfaces. No new CLI flags, matching how other nested sub-config (router.*, trunk.*) is set via config.toml rather than per-field flags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 10:57:47 +02:00
lars	f301fd98d2	Make HistoryEncoder a pluggable registry, like Router/Objective (gitea #35) ... Stage2Autoregressive.init_history_cache and .history_step both isinstance-checked self.history_encoder against AttentionHistory to decide whether to use its real incremental-cache methods or a no-op fallback, so a third history type couldn't be added without editing Stage2Autoregressive itself. The two-value "markov"/"attention" enum was also independently hardcoded in three places (Stage2Autoregressive's own validation, config.py's validate_config, and AutoregressiveConfig.from_dict's default). Mirrors the Router (giant/model/routers.py) and Objective (giant/model/objectives.py, gitea #32) pattern: HistoryEncoder now declares working O(1) init_cache/step defaults (init_cache -> None, step -> one forward() call), so every registered history type satisfies the incremental interface without opting in; AttentionHistory overrides both with its real KV-cache versions since its forward() needs the full prefix. Added HISTORY_REGISTRY/register_history/build_history, registered "markov" and "attention", and deleted both isinstance checks in models.py. Per user decision during planning, config.py's validate_config now imports HISTORY_REGISTRY and checks membership dynamically instead of keeping its own hardcoded tuple, making the registry the single source of truth end to end (verified no import cycle: config.py had no prior dependency on giant.model, and giant.model.history has none on giant.config). No config-schema change and no checkpoint impact — this is a pure internal-interface refactor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 10:35:46 +02:00
lars	f8722e347e	Add an Objective registry for the flow/ddpm/wgan generator choice (gitea #32) ... generator ∈ {"flow", "ddpm", "wgan"} was tested as a bare string in ~45 sites across models.py, sample.py, builders.py, trainers.py, and stage2_inputs.py, each independently re-deriving one of five consequences of the choice (needs a time embedding? what does the trunk take as input? is the type slice folded into the trunk output? which sampler? which loss?). giant/model/objectives.py adds an Objective ABC + OBJECTIVE_REGISTRY + build_objective factory, mirroring routers.py's Router pattern, and every bare-string site now goes through it (needs_time, is_adversarial, folds_type_slice, trunk_in_dim, build_schedule, stage1_loss/stage2_loss). Per discussion: FlowDDPMStageTrainer and WGANStageTrainer stay separate classes rather than merging into one StageTrainer as the issue's sketch proposed — their training loops are genuinely different shapes (single loss vs. dual G/D step with gradient penalty/n_critic/ST-Gumbel), and trainers.py is the least-covered-by-fast-tests part of the codebase, so a full merge was judged out of proportion to this issue's risk budget. FlowDDPMStageTrainer's own loss dispatch (flow vs ddpm, one-shot vs AR) does move onto the objective, so a future non-adversarial objective (rectified flow, consistency distillation) is still a one-file, zero-trainer-edits addition. No config-schema change — stage{1,2}_model.generator stays the persisted string, just looked up in the registry instead of string-compared. An unrecognized generator value now fails fast with a clear ValueError instead of silently falling through some bare-string checks and not others (same behavior build_router/build_trunk already have for their own type keys). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 10:10:58 +02:00
lars	0f95e0eaae	Make ResBlock's conditioning-injection mechanism selectable (gitea #34) ... ResBlock injected conditioning exactly one way — h = linear1(h) + cond_proj(cond), a conditional bias, the weakest standard option for a model whose entire job is to be conditional. Adds BLOCK_REGISTRY (giant/model/layers.py), mirroring the TRUNK_REGISTRY/ROUTER_REGISTRY registry+factory idiom (gitea #33), with two new drop-in alternatives: FilmResBlock (per-channel scale+shift modulating the norm output, zero-init so conditioning has no effect at construction) and AdaLNResBlock (DiT-style AdaLN-Zero — the norm's own affine is replaced by a conditioning-derived scale/shift, plus a zero-init gate on the residual branch, making the block the exact identity function at init). Selected per stage via a new stage{1,2}_model.trunk.block_conditioning config leaf ("add" | "film" | "adaln", default "add"), threaded through build_trunk/build_expert_body/RoutedTrunk and the three stage model constructors. Default stays "add" and ResBlock's body is unchanged, so existing configs/checkpoints are bit-identical to before this change. Decided during planning: the new field lives on the existing TrunkConfig rather than a new top-level block/blocks config section; the WGAN CriticModel (which builds its own ResBlock stack outside TRUNK_REGISTRY) and the issue's mentioned blocks.norm/blocks.activation axes are both left out of scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 09:41:50 +02:00
lars	f3f7645bf7	Make trunk architecture selectable via a registry (gitea #33) ... build_trunk hardcoded exactly two shapes (MonolithicTrunk/RoutedTrunk), chosen only by whether a Router was built, with no way to select a different trunk body architecture at all. Deviates from the issue's literal proposal (a TRUNK_REGISTRY choosing between "resmlp"/"moe" trunk shapes): during planning, decided that the trunk *body* architecture and whether it's *mixed* are orthogonal, so the registry (TRUNK_REGISTRY/register_trunk/build_expert_body in giant/model/trunks.py) holds expert bodies only (today: "resmlp", ExpertTrunk's existing input_proj -> ResBlock stack -> out_proj). Routing stays exactly router.enabled/n_experts, untouched — a future transformer body gets a mixture variant for free (trunk.type = "transformer" + router.enabled = true) instead of needing a separate registry entry per (body x routed/not) combination. MonolithicTrunk is deleted; the unrouted case now returns the registry-selected body directly, preserving today's exact state-dict keys (trunk.input_proj.* etc., not trunk.experts.0.*) — required both for existing non-routed checkpoints and because _legacy.py's migrate_legacy_state_dict already assumes that flat layout for a v0.2 checkpoint. New config leaf only: stage{1,2}_model.trunk.type: str = "resmlp" (TrunkConfig). hidden_dim/n_res_blocks/dropout stay where they are today. Nothing about router.enabled, config.migrate_config, _legacy.py, or the CLI's --router flags changes — a v0.2-migrated config gets trunk.type = "resmlp" automatically, reproducing current behaviour exactly. No CLI flag added (matches the config.toml-only precedent set by autoregressive.history/particle_type.target/n_sec.mode). No transformer body and no "none"/"linear" body (gitea #45) in this change. Full design rationale recorded on gitea #33 and #45 before implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 09:16:55 +02:00
lars	f505fe7f22	Skip router auxiliary loss compute when their lambda is 0 (gitea #31) ... FlowDDPMStageTrainer._compute unconditionally called router.balance_loss/classify_loss/entropy_loss whenever a router existed, then only added each term into total if its lambda was > 0 -- so every routed run paid for balance_loss/entropy_loss's extra router.gate(...) forward passes even at the default lambda_balance = lambda_proc = lambda_entropy = 0.0 (the exact config the failed 2026-07-22 router benchmark ran). Guard each computation on the same > 0 condition that already guarded the addition, matching WGANStageTrainer's cost structure which has no router-loss block at all. total's value is unchanged either way. Added a test that spies on the router's three loss methods and checks call counts both at lambda=0 (must be skipped) and lambda>0 (must still run, so the guard doesn't suppress the real path). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 16:18:19 +02:00
lars	32aa5a5f92	Decouple secondary-species vocabulary from conditioning.particle.emb_dim (gitea #29) ... conditioning.particle.emb_dim and stage2_model.particle_type.target="onehot"'s class count were silently the same number everywhere (pipeline.py's PDG top-N map build, Stage2OneShot/Stage2Autoregressive's type head, StageSpec's training loss width, the checkpoint's shared pdg_topn_map), fixing the secondary-species vocabulary at whatever width the unrelated physical-conditioning MLP happened to use — the exact vocabulary the v0.3.0 pivot exists to fix. Adds stage2_model.particle_type.n_classes (default 0 = inherit conditioning.particle.emb_dim, preserving today's behavior and every existing checkpoint) and a single resolve_type_n_classes helper used everywhere the coupling used to be implicit. Splits the checkpoint's shared pdg_topn_map into a conditioning-only pdg_topn_map and a new sec_type_topn_map, built independently through the existing (axis, n_classes)-keyed setup cache (no extra scan when they still resolve to the same N) and threaded through giant predict/giant rollout's decode path. A checkpoint with no sec_type_topn_map key (pre-#29) falls back to reusing pdg_topn_map, reproducing the old shared behavior exactly. Decided with the user during planning: commit directly on this branch; represent the split as an additive sec_type_topn_map checkpoint key rather than conditionally reusing pdg_topn_map; build the two top-N maps independently rather than the issue's proposed build-at-max-and-slice, since the setup cache already avoids redundant scans across runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 16:11:14 +02:00
lars	899ca3a7d5	Validate stage2_model.autoregressive.order in validate_config (gitea #30) ... order was documented as single-valued ("energy_desc" only, placeholder for a future alternative ordering) but validate_config only checked its siblings history/teacher_forcing, so e.g. order = "energy_asc" was silently accepted and trained as if it were energy_desc. Add the missing check alongside the other two, gated the same way (only meaningful under stage2_model.decoder = "autoregressive"). Also updates the stale reason string on the pre-existing _KNOWN_UNUSED allow-list entry for this key in tests/test_config_consumed_keys.py, since half of it ("validate_config ... never [checks] order") is no longer true after this fix — the key stays allow-listed because validate_config itself isn't in that test's build/train/rollout consumer whitelist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 15:40:47 +02:00
lars	da717971b6	Honour wgan.critic_hidden_dim/critic_n_res_blocks in build_critics (gitea #28) ... build_critics always sized a WGAN critic off the generator's own hidden_dim/n_res_blocks, silently discarding the documented 0=inherit sentinel on stage{1,2}_model.wgan.critic_hidden_dim/critic_n_res_blocks (the same convention critic_lr already honoured). Now both keys are read with the 0 -> inherit fallback, and stage-scoped-only CLI flags (--stage{1,2}-critic-hidden-dim/--stage{1,2}-critic-n-res-blocks) are added -- no shared alias, since critic sizing is an architectural per-stage knob like --hidden-dim/--n-res-blocks, not a shared training hyperparameter like --n-critic/--gp-weight/--noise-dim/--critic-lr. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 15:36:26 +02:00
lars	c3fc768b40	Reject stage2_model.stage1_context = 'sampled' as unimplemented (issues.md Issue 1) ... trainers.py unconditionally trains stage 2 against the ground-truth stage-1 output (stage1_ctx = x1_s1.detach()), but 'sampled' was accepted by validate_config, stored in config.toml and the checkpoint's model_config, and silently trained identically to 'truth' — mislabeling every downstream artifact for a run launched with --stage2-stage1-context sampled. Mirrors the existing stop_token validate_config pattern. User chose the immediate fix (reject loudly) over the proper fix (actually implement sampled context), which is scoped to Issue 16. Also updates the _KNOWN_UNUSED reason for stage2_model.stage1_context (added by Issue 5's consumed-keys audit) to reflect that the value is now rejected rather than silently accepted, and drops the now-invalid --stage2-stage1-context sampled case from test_stage2_only_knobs (a full CLI invocation) — that flag's plumbing is still covered at the overrides-dict level by test_overrides_from_flags_stage2_only_knobs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 14:57:21 +02:00
lars	a4b5a6c3bf	Add consumed-keys audit test (issues.md Issue 5) ... validate_config_keys only checks that a config key is declared in DEFAULT_CONFIG, never that anything reads it — the gap that let Issues 1, 2 and 4's dead keys (stage1_context, wgan.critic_hidden_dim/critic_n_res_blocks, autoregressive.order) slip through silently. tests/test_config_consumed_keys.py walks every DEFAULT_CONFIG leaf path and asserts each is either found (via AST scan for attribute access, dict-key-shaped string constants, or constructor/ function parameter names — the last needed because Router subclasses receive their config via **kwargs filtered by signature) in a fixed whitelist of build/train/rollout consumer files, or explicitly recorded in _KNOWN_UNUSED with a reason. A second test asserts the allow-list has no stale entries, so fixing Issue 1/2/4 will force removal of the corresponding allow-list line rather than let it silently outlive the bug. The whitelist is intentionally narrower than "anywhere in giant/": scanning the whole package produces false negatives from unrelated identifier collisions (e.g. router_gating.py's unrelated `order` parameter would make autoregressive.order read as consumed). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 14:42:47 +02:00
lars	81eb14d75c	Move scripts/ to giant/tools/ (issues.md Issue 9) ... `scripts` was published as a top-level distribution package, colliding with one of the most generic names in the Python ecosystem and shadowable by a stray scripts/ dir on the portal machines' shared /work/lbogner. Move it under the giant namespace; the dwarf command name is unchanged, only the Python import path and file location move. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:31:47 +02:00
lars	a4f4cba58b	Type the data/model/training batch contracts with NamedTuples (issues.md Issue 7) ... build_features (transforms.py) now returns StepFeatures and StreamingStepsDataset (dataset.py) now yields StepBatch, both NamedTuples with the same field order as the tuples they replace, so ty can catch a dropped/added field at every consuming call site instead of a silent positional-tuple mismatch. Converted the unreadable throwaway-heavy unpacks in cli.py, pipeline.py, validate.py, and dataset.py to named attribute access; gave the WGAN path's derived 5-element batch its own _Stage2RealFakeBatch NamedTuple; updated the two test batch-construction helpers to build real StepBatchs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:11:58 +02:00
lars	e6261cea03	Unify the two v0.2->v0.3 migration surfaces (issues.md Issue 6) ... giant/config.py:migrate_config (config.toml) and giant/model/network.py:_migrate_legacy_model_config (checkpoint model_config) independently hand-maintained the same v0.2 facts and an identical router expert-sizing rejection. Extract the shared knowledge into a new leaf module, giant/_migration.py (V02_MODEL_KEY_TO_STAGES, V02_FIXED_FACTS, reject_legacy_router_expert_sizing), consumed by both. Also replace NSecConfig's legacy-only, nullable legacy_owner sentinel (living in an extra: dict catch-all) with a normal, always-set owner: str = "stage2" field, so build_models reads one concrete two-valued key instead of branching on a legacy marker. Record in CLAUDE.md that v0.2 checkpoint-loading support has no expiry decided yet, since /ceph still holds pre-v0.3.0 checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 09:56:22 +02:00
lars	818c380fd0	Extract predict/rollout's duplicated inference bootstrap into giant.checkpoint_io (issues.md Issue 5) ... giant predict and giant rollout each carried a ~65-line, independently drifting copy of "load checkpoint -> validate -> resolve conditioning axes -> restore normalizers/vocab maps -> build models -> load weights", plus a third partial copy of _conditioning_axes in analysis/router_gating.py. A silent divergence there doesn't crash, it makes the two commands run different physics from the same checkpoint with no test coverage anywhere along that path. giant/checkpoint_io.py now holds the single implementation: load_for_inference() + an InferenceContext dataclass, raising CheckpointCompatibilityError (verbatim message text preserved) instead of calling typer directly, so it can be unit-tested and imported from non-Typer code. router_gating.py's load_router imports conditioning_axes from it lazily, keeping its "no torch at module scope" contract intact. Adds 17 direct unit tests for load_for_inference/conditioning_axes/stage_cfg plus CLI smoke tests confirming the error surfaces as typer.Exit(1) through predict and rollout — previously zero coverage on this path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:31:14 +02:00
lars	2bfb1ab056	Extract giant train/new-run's CLI override mapping into a table-driven function (issues.md Issues 3 & 4) ... train()'s ~140-line hand-written flag->config translation (three different ad hoc "more specific flag wins" patterns) and new_run()'s near-verbatim copy are replaced by a shared FlagSpec/FLAG_SPECS table and overrides_from_flags() in config.py, reused by both commands. This makes the override/precedence logic directly unit-testable without CliRunner, closing coverage gaps that had zero tests (e.g. --emb-dim/--conditioning dual-axis fan-out, three of four WGAN knob legs, --stage2-generator overriding --mode, router's stage1-only asymmetry). No CLI flags, help text, or precedence semantics changed -- `giant train --help`/`giant new-run --help` are byte-identical before and after, and all previously-passing CliRunner tests still pass unmodified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:04:49 +02:00
lars	01acbfed61	Add unknown-key validation to config.toml merge (issues.md Issue 2) ... A typo like `n_res_block` for `n_res_blocks` previously merged cleanly, passed validate_config, and silently trained a model that didn't match config.toml's documented settings. merge_cli_overrides now rejects any key not present in DEFAULT_CONFIG's schema via validate_config_keys, with a did-you-mean suggestion, while still allowing the genuinely dynamic composed-router axis keys and centers_init. Checkpoint model_config loading is untouched, so old checkpoints keep loading. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 14:47:29 +02:00
lars	9bf5874308	Make config dataclasses the single source of truth for DEFAULT_CONFIG ... DEFAULT_CONFIG and build_models/build_critics/StageSpec.from_config's inline .get(key, default) fallbacks had already drifted: two keys (stage2_model.decoder, stage2_model.particle_type.target) resolved differently depending on whether a config dict came from merge_cli_overrides (fully populated, correct) or was hand-built and partial (fell back to stale v0.2-shaped literals). Introduce frozen dataclasses (GiantConfig and its nested blocks) in giant/config.py as the actual single declaration of every default; DEFAULT_CONFIG is now generated from them instead of hand-maintained, and build_models, build_critics, and StageSpec.from_config consume the dataclasses instead of duplicating literal fallbacks, so this class of drift can't recur. Router/n_sec sub-blocks keep an `extra` catch-all for their genuinely dynamic keys (composed-router axes, runtime-seeded centers_init, legacy_owner). Fixing the fallback surfaced the same latent bug in two existing partial-config callers that had been silently depending on it: a test fixture in test_train.py and scripts/warm_setup_cache.py's minimal cfg (now merged against DEFAULT_CONFIG instead of hand-rolled, closing the gap for good). See issues.md Issue 1. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 14:35:14 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	9ce7b32324	Fix test_render_all_run_gallery_invokes_subprocess clobbering LaTeX's own subprocess.run ... render_mod.subprocess is the stdlib subprocess module itself, not a copy — patching .run unconditionally also intercepted the real subprocess.run calls matplotlib's texmanager makes to compile LaTeX during savefig, so those returned the test's fake return value instead of a real CompletedProcess and crashed with AttributeError: 'NoneType' object has no attribute 'stdout' on any environment where render_all runs before the gallery call (i.e. everywhere but this dev machine's warm state that happened to mask it). Only intercept the "gallery generate" call now; everything else passes through to the real subprocess.run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:56:36 +02:00
lars	82772e4e09	Add render.py coverage: figure params, router diagnostics plots, gallery/condor glue ... render.py was at 58% coverage — the module's plotting dispatch (router_gating, router_share, unavailable) and glue logic (_figure_params/_figure_params_v2, _plot_metadata, render_all's gallery subprocess call, render_run's condor RunMeta wiring) had no tests at all. Brings it to 100%: pure-function unit tests for the v0.2/v0.3.0 figure-param branches and _plot_metadata, real LaTeX-rendered fixtures for the previously-untested plot kinds and a 4-group grouped_hist (exercises the hidden-leftover-axis branch), and mocked subprocess/condor calls to isolate render_all/render_run's own logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:47:31 +02:00
lars	24445b7427	Add coverage for router-center seeding, geometry batch reader, material topN cache, and setup-cache corruption paths ... Closes the highest-value coverage gaps found via pytest-cov: pipeline.py's EnergyRouter quantile-seeding (the roadmap's flagged fix for the failed MoE rollout benchmark) had zero coverage, geometry.py's real parquet-batch reader was always mocked, the material top-N-map cache-hit branch was untested (only pdg's was), and setup_cache.py was missing malformed-cache-body and unknown-axis error paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:38:04 +02:00
lars	878e9ddca3	Delete docs/v0.3.0-design.md and strip all references to it ... The design doc and its followups doc are no longer needed as a live reference now that the v0.3.0 redesign is implemented — comments and docstrings across the codebase cited it extensively (file path, "design doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design rationale. Removed docs/ and edited every citing comment/docstring to drop the now-dangling reference while keeping the substantive explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its trailing pointer to the deleted file. Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or "§N.N" references (repo-wide grep); ruff and ty clean; full test suite on the heaviest-touched modules (network, sample, rollout, migration, config, train) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:19:02 +02:00
lars	d3271bc798	Silence the fork-safety warning from num_workers>0 pipeline tests ... The two DataLoader-num_workers quota tests are the only ones in the file that leave num_workers>0, so they're the only ones that actually spawn forked worker subprocesses under pytest's multi-threaded process and hit Python's fork-safety DeprecationWarning. The thing under test is just the pre-flight quota-check message, emitted before the DataLoader is built. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 09:48:58 +02:00
lars	8019a80563	Refactor train.py into giant/training/ around a metrics collector ... Every metric name used to exist in four places: the dict keys each StageTrainer returned, the hardcoded _metrics_fields() column list, the ~110-line metrics_row assembly in train(), and the tqdm/summary formatting. The two had to be kept in exact correspondence by hand or csv.DictWriter would raise. Each metric is now declared once, as a MetricSpec on the trainer that computes it. MetricsCollector derives the CSV header and W&B payload from those declarations and owns all accumulation, so train() no longer carries a running sum, and every isinstance(tr, WGANStageTrainer) branch is gone — replaced by four trainer hooks (batch_loss, summary, val_objective, supports_val_loss). giant/train.py (1875 lines) becomes giant/training/: trainers.py StageSpec + shared StageTrainer base + the two subclasses metrics.py MetricSpec, MetricsCollector stage2_inputs.py the pure AR/teacher-forcing tensor helpers, moved verbatim loop.py train() (225 lines, was ~514) + graceful shutdown checkpoint.py build/load, lifted out of train()'s closures The trainers shared ~15 identical constructor arguments and copy-pasted their cosine-warmup lambda, EMA setup, state_dict/load_state_dict, resume_lr and train_mode/eval_mode. StageSpec resolves one stage's config once (constructors go from 24 and 22 keyword arguments to (spec, model, device)), the base class holds the rest, and build_stage_trainers drops from ~100 lines to 15. Metric columns are renamed to a uniform stage/split/metric scheme (stage1/train/loss, stage2/train/d_loss, stage1/lr, stage1/router/entropy, val/loss, ...). Old metrics.csv files and W&B history are not comparable. The checkpoint format is unchanged. BEHAVIOR CHANGE — WGAN best-checkpoint selection. The old code meant to score a WGAN stage on its marginal KL, but the guard `{n: kl for n in wgan_names if n not in val_loss_per_stage}` could never fire: val_loss_per_stage was pre-seeded with 0.0 for every stage, so a WGAN stage contributed a flat 0.0 and the KL was written to metrics.csv without ever influencing best.pt. val_objective now returns it as intended. On the test harness's default flow+wgan config val_loss went from 2.182 (stage 1 only) to 15.137 (stage 1 + KL 12.954), and which epoch won changed. Runs before this commit picked their best checkpoint on the non-adversarial stages alone. Written up in docs/v0.3.0-followups.md. Verified: 699 tests pass; ruff, ruff format and ty clean. Baseline-vs- refactor metrics.csv compared across five configs (flow+wgan, AR+onehot, routed, both-flow, AR-flow) — every comparable value bit-identical except val/loss where the fix applies. Resume appends without a duplicate header and reproduces a HEAD worktree's per-epoch losses and LRs exactly across the resume boundary. A refactored last.pt loads through cli.py:_load_model_weights in both raw and ema modes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-07 17:03:20 +02:00
lars	da7cde3ef9	v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies ... Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 16:12:58 +02:00
lars	200c6d243b	v0.3.0 step 7: AttentionHistory (KV-cached) + scheduled/never teacher forcing ... AttentionHistory (giant/model/network.py) adds causal self-attention over the emitted-secondary prefix as the alternative to MarkovHistory, with a parallel forward() for training and an init_cache()/step() KV-cache path for sample.py's per-slot AR inference loop, wired into Stage2Autoregressive via history="attention". giant/train.py adds _stage2_tf_prob and _assemble_stage2_ar_inputs_scheduled, mixing ground-truth history with a detached sample_secondaries_ar self-sample per slot so teacher_forcing="scheduled"/"never" close the train/inference gap teacher_forcing="always" always avoided; wired into both stage-2 AR trainers. config.py's validate_config no longer rejects these two previously unimplemented schema values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 13:15:56 +02:00
lars	93b19911f8	v0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode ... - giant/sample.py: fix every sampler's call convention against Stage1Model/Stage2OneShot's actual forward signatures (was still calling model(x, t, cond_cont, cond_cat) positionally); add sample_secondaries_ar (free-running AR loop, unsnapped history feature) and sample_stage1/sample_stage2/resolve_n_sec dispatch helpers that read each stage's generator_kind/decoder off the model instance itself. - giant/particles.py: decode_topn_class (argmax + other_policy) and decode_embedding_nearest (L1-snap + distance) turn a secondary's "onehot"/"embedding" type prediction into a concrete PDG. - giant/rollout.py: decode_secondary_identity routes all three particle_type.target values to real mass/charge; per-stage generator dispatch (drops the single shared `mode` string, adds ddpm support); L1DistCollector accumulates the §11.3 embedding-distance diagnostic. - giant/cli.py: drop the onehot/embedding-target rejection gate (narrowed to the still-unimplemented conditioning.particle/material.type=onehot axis); fix the dead model_cfg.get("mode") bug in predict/rollout. - giant/analysis/: new type_embedding_l1_distance PlotSpec, wired through the rollout YAML sidecar (no live-model call needed, unlike router_gating -- the histogram is already pre-aggregated at rollout time). - Un-xfail every test that was blocked on this step (test_rollout.py, test_flow.py, test_wgan.py, test_phase2.py, test_router.py, test_validate.py); add test_sample.py, test_type_embedding_distance.py. Known follow-up: giant/validate.py still unpacks the training val-batch as a stale 6-tuple and doesn't use the new per-stage dispatch, so marginal validation during training degrades gracefully with a warning rather than working -- not in this step's scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 10:37:57 +02:00
lars	c9d255b1c5	v0.3.0 step 5: Stage2Autoregressive (history=markov) + §11.4 grad instrumentation ... Replaces the Stage2Autoregressive stub with a real per-token secondary decoder: MarkovHistory summarizes the previous secondary, remaining-energy fraction and slot index round out the per-token conditioning, and the existing Trunk/MonolithicTrunk/RoutedTrunk machinery is reused unchanged by batching all K_MAX tokens together under teacher forcing (one parallel pass, no new trunk code). build_models/build_critics wire it in; the WGAN critic stays whole-sequence, so build_critics needs no AR-specific path. train.py's FlowDDPMStageTrainer/WGANStageTrainer gain a decoder branch, sharing optimizer/EMA/checkpoint machinery with the one-shot path. _assemble_stage2_real is now defined in terms of the new unflattened _assemble_stage2_ar_target helper, removing a near-duplicate branch. Also lands the §11.4 differentiability validation-obligation instrumentation (trunk-gradient norm from the particle-type slice vs. the continuous slices, for generator=wgan + particle_type.target=onehot) via backward hooks in _relax_onehot_type_slice, decoder-agnostic and surfaced as two new metrics.csv columns. This also fixes the standing regression where any config not explicitly overriding decoder="one_shot" crashed at build_models, since stage2_model.decoder defaults to "autoregressive" — confirmed by removing tests/test_pipeline.py's now-stale override so the default config runs end-to-end against real synthetic data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 09:36:49 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	9112e845e0	v0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite ... Replaces train.py's single global training loop with a StageTrainer hierarchy (FlowDDPMStageTrainer, WGANStageTrainer) — one per active stage, each owning its own optimizer/LR schedule/EMA and reading only the shared batch tuple (stage 2 always teacher-forces on the ground-truth x1_s1, so stages never need each other's output at train time). Supports every stage1/stage2 generator combination, including the design doc's headline mixed case (stage1=flow + stage2=wgan) and its reverse, plus stage1-only/stage2-only ablation runs, routed+gumbel stages, and checkpoint save/resume. metrics.csv/wandb logging are stage-prefixed. validate_marginals calls are guarded with a one-time warning and a Wasserstein-magnitude fallback for wgan best-checkpoint selection, since giant/sample.py still assumes stage1 always owns n_sec_head (decision 1 moved it to stage 2 by default) — deferred to design doc step 6, not silently papered over. pipeline.py's run_setup_stage/run_train_job now read the new nested config directly; the dangling resolve_expert_dims call and the --mode wgan --router rejection are both gone (routed WGAN works). cli.py's train/new-run build correctly-shaped config overrides (architecture flags -> stage1_model only per the approved decision; --mode/--n-critic/--gp-weight/--critic-lr broadcast to both stages, matching migrate_config's own precedent and avoiding a regression on the common --mode case); predict/rollout's dangling build_models tuple-unpack is fixed; new-run now tags config_version, fixing a bug where a re-loaded v0.3 config.toml would have been silently corrupted by migrate_config mistaking it for v0.2. config.py's validate_config rejects mixed particle/material conditioning types for now (ConditionEncoder supports it, the data pipeline in giant/data/transforms.py doesn't yet). analysis/render.py and router_gating.py handle both the new nested model_config shape and legacy flat checkpoints. scripts/warm_setup_cache.py updated for run_setup_stage's new signature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 11:31:49 +02:00
lars	9ce55e5013	v0.3.0 step 2: network.py refactor to composable stage models ... Decomposes the ten permutation classes in giant/model/network.py into the reusable parts from docs/v0.3.0-design.md §5: ConditionEncoder (now independently configurable per particle/material axis), ContextAdapter, Trunk/MonolithicTrunk/RoutedTrunk/ExpertTrunk, and the stage classes Stage1Model/Stage2OneShot/CriticModel (Stage2Autoregressive stubbed, raises NotImplementedError until step 4/5). build_models/build_critics now return a dict keyed by stage and accept the new nested config shape, with routed WGAN reachable for the first time (the old --mode wgan --router rejection is gone) and stage2_model.router.tie_to_stage1 sharing a literal Router instance. A v0.2 checkpoint's flat model_config auto-migrates via _migrate_legacy_model_config + migrate_legacy_state_dict, preserving the n_sec_head's attachment to Stage1Model (legacy_owner="stage1", design doc §4.1). tests/test_migration_v02_v03.py proves this bit-identical against a frozen v0.2 snapshot (tests/legacy/network_v02_snapshot.py) for both flow and wgan, both conditioning modes. scripts/check_migration_v02_v03.py is the real-checkpoint counterpart for a portal machine with /ceph access. giant/model/schedule.py's flow-matching/DDPM loss helpers are updated to the new model-call convention (t as a keyword). giant/sample.py, giant/rollout.py, and giant/validate.py are not yet updated (deferred to design doc step 6) — their exercising tests are marked xfail with that reasoning rather than silently broken. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:55:29 +02:00
lars	eb6dd27406	v0.3.0 step 1: new nested config schema, v0.2 migration shim ... Replace the single global train.mode + [model] block with the four top-level blocks docs/v0.3.0-design.md specifies ([conditioning], [stage1_model], [stage2_model], [train]), so Stage 1 and Stage 2 can run independent generative objectives and Stage 2 can train standalone. - migrate_config translates old config.toml/checkpoint dicts on load, so nothing on /ceph goes dead; loudly rejects non-zero expert_hidden_dim/expert_n_blocks, which v0.3.0 no longer supports. - merge_cli_overrides/save_config generalize from one hardcoded nesting level (model.router) to arbitrary recursive depth. - default_out_dir_name candidates move to dotted paths against the new schema, with per-stage router/generator discriminators. - validate_config adds cross-block checks the per-block schema can't express (particle_type.target=embedding needs a matching conditioning mode, tie_to_stage1 needs an active stage 1, etc). - resolve_expert_dims is deleted (experts always inherit the stage's hidden_dim/n_res_blocks now) — pipeline.py/cli.py callers are left dangling on purpose, to be updated in the network.py/train.py steps that follow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:15:20 +02:00
lars	803aae364e	format: Format tests/test_condor.py according to ruff styling	2026-08-03 14:57:45 +02:00
lars	057d637080	Fix test_write_submit_requires_synced_venv for active-venv resolution ... The venv-detection change in condor.py now checks for a giant binary next to sys.executable before falling back to repo_dir/.venv/bin/giant, and the test's own pytest venv has one — so the expected FileNotFoundError never fired. Monkeypatch sys.executable to a nonexistent path so the test exercises the fallback with neither location populated.	2026-08-03 14:52:41 +02:00
lars	ca3a2a3462	Fix CLI/tooling robustness gaps and dedupe the Conditioning enum ... - run_create_manifest gains --force; it previously overwrote an existing manifest (including holdout.manifest, which check_holdout_overlap exists specifically to protect) with no warning or backup on a second run. - _git_user_name only caught OSError, not subprocess.TimeoutExpired (a SubprocessError, not an OSError) — a slow/loaded shared portal machine could crash `dwarf bump-gen`/`bump-schema` instead of degrading to by=None as intended. - `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when the requested count exceeds ~1/4 of the machine's CPUs, matching the same shared-machine etiquette check added to giant train in the previous commit. - The Conditioning enum was independently redefined in both giant/cli.py and scripts/dwarf.py; moved to a single giant.config.Conditioning both now import, removing the drift risk of a third conditioning mode being added to one but not the other. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:49:09 +02:00
lars	ad1b8e7835	Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline ... - prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:43 +02:00
lars	ad0341a9d4	Fix training-loop checkpoint/resume and WGAN bugs ... - Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of in-progress weights before exiting mid-epoch — it previously broke out of the epoch loop before reaching the checkpoint-save block, contradicting its own printed "saving a checkpoint" message and losing all progress since the last completed epoch. Checkpoint-dict construction is factored into a shared _build_checkpoint() helper used by both the mid-epoch and end-of-epoch save paths. - WGAN LR-schedule steps_per_epoch used the wrong denominator (n_critic + 1 instead of n_critic), causing the schedule to exhaust early and LR to floor to 0 before training completed. - --critic-lr override was silently dropped on WGAN --resume (only the generator optimizer's LR was made authoritative again after load_state_dict; optimizer_d's was not). - WGAN secondary gradient-penalty forced x_hat/grad to zero for fully-masked rows (n_sec == 0, common in a shower), adding a constant ~1.0 bias into the batch-mean GP term; such rows are now excluded from the mean. - run_train_job warns (never blocks) when --num-workers exceeds ~1/4 of the machine's CPUs, per this repo's shared-portal-machine etiquette (see CLAUDE.md's Compute environment section). Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:22 +02:00