120 Commits

Author	SHA1	Message	Date
lars	c4b12b5e7a	Pass ConditioningAxisConfig/ParticleTypeConfig themselves instead of raw dicts (gitea #38) ... build_models/build_critics parsed model_config into frozen dataclasses (ConditioningConfig, Stage2ModelConfig, ...) but then threw the parsed sub-objects away and passed the original raw dicts (conditioning["particle"], s2_spec.particle_type.to_dict()) down into ConditionEncoder/StageModel/etc, which re-read them with their own hardcoded .get(key, default) fallbacks — each an independent copy of a fact the dataclass already stated once. Worst instance: giant/training/trainers.py:236 converted an already-parsed ParticleTypeConfig back into a dict for no reason. Threads ConditioningAxisConfig (particle_cfg/material_cfg) and ParticleTypeConfig (particle_type_cfg) as the actual dataclass instances through every signature that used to type them dict: ConditionEncoder, StageModel/CriticModel, resolve_type_n_classes/stage2_type_dim/ stage2_trunk_sec_dim, giant/model/builders.py, giant/sample.py, giant/training/stage2_inputs.py, giant/training/trainers.py (StageSpec/ StageTrainer), giant/pipeline.py, giant/rollout.py, giant/validate.py — so ty now catches a misspelled field instead of it silently falling back. No config-schema change: config.toml/checkpoint model_config keep the same nested-dict shape; only what happens after the existing X.from_dict(...) parse changes. User-confirmed scope decision: both axes (particle_cfg/material_cfg and particle_type_cfg), not just the more heavily-duplicated particle_type_cfg axis, and not stopping at the two most literal parse-then-discard round trips — matching the issue's own proposal. Preserved-default decision: StageModel's particle_type_cfg=None sentinel (hit only by direct/test construction — build_models always passes an explicit particle_type) still resolves to ParticleTypeConfig(target= "physical"), not ParticleTypeConfig()'s own target="onehot" config-file default — switching it would have silently grown an unused, gradient-less type_head on every test that constructs Stage2OneShot/Stage2Autoregressive without particle_type_cfg=, breaking their "every param has a grad" checks. New tests in tests/test_network.py: ConditionEncoder/StageModel store the exact ConditioningAxisConfig/ParticleTypeConfig instance passed in (identity, not just equality) — no internal dict round-trip — and build_models's output carries real dataclass instances end to end, not the plain dicts it produced before this fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 15:03:55 +02:00
lars	1a3c907571	Give Stage1Model/Stage2OneShot/Stage2Autoregressive a shared StageModel base (gitea #39) ... Stage1Model, Stage2OneShot and Stage2Autoregressive each independently implemented ~90 near-identical lines of __init__ scaffolding: build-or-share cond_enc, particle_type_cfg normalisation, objective -> time_emb -> merged_cond_dim -> build_trunk, and the n_sec_head/type_head classifier heads (plus their identical RuntimeError guards). Now unblocked by #33 (trunk registry), #34 (block-conditioning registry) and #36 (build_mlp_head), which settled what belongs in the shared base. Adds StageModel(nn.Module) owning all of that: __init__ builds/shares cond_enc and normalises particle_type_cfg; _build_trunk_and_heads, called by each subclass after it sets up its own conditioning-assembly modules (cond_enc alone for Stage1Model, a context-fusion path for the two Stage2 classes), builds the objective/time embedding/trunk and the n_sec_head/type_head guarded by the shared _require_n_sec_head/ _require_type_head (Stage1Model overrides the n_sec guard since its message points at stage 2, not stage 1). Public __init__ signatures, attribute names, and forward/predict_* behaviour are unchanged. Verified with a pre/post state_dict-key-set diff against the pre-refactor classes (bit-identical) before writing this commit, plus new parametrized tests pinning each class's state_dict key set and the generator -> time_emb contract the base now owns. tests/test_migration_ v02_v03.py's existing bit-identical old-vs-new forward comparison and the rest of tests/test_network.py's per-class coverage pass unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 14:37:58 +02:00
lars	4692cee699	Give the cond_cat/cond_cont column layout one owner (gitea #37) ... The conditioning arrays' column order was written down three times — twice in giant/data/transforms.py (build_cond_features and build_features each built cond_cont and cond_cat from scratch) and again in giant/model/encoders.py (cat_col_layout, plus hand-written COND_DIM_BASE + PARTICLE_PHYS_DIM slicing in ConditionEncoder). The three were held in sync only by parallel comments, so a wrong column order produced silently mis-indexed features rather than an exception. The drift had already happened, twice, both times in build_features: - 5b63dfd added per-axis vocab-lookup strictness (an out-of-vocab pdg/material must not KeyError under "physical"/"onehot", where the index is never read) to build_cond_features only. - _cond_normalizer_transform's legacy-normalizer padding, which keeps a pre-physical-conditioning 8-wide cond normalizer loadable, was likewise only wired into build_cond_features — so `giant predict` on such a checkpoint died with a broadcast error. New giant/cond_layout.py holds a frozen CondLayout built from the (particle, material) mode pair, exposing named cond_cont slices (base/particle_phys/material_phys) and cond_cat columns (PDG_COL/MAT_COL/particle_topn_col/material_topn_col/cat_dim). Both builders now share one _build_cond_arrays, ConditionEncoder reads its slices off the same object, and PdgRouter/ProcessRouter use the named dense-vocab columns instead of literal 0/1. CondLayout also absorbs the two duplicated axis-type validations, keeping their message text verbatim. Decisions taken while planning: - Scope is CondLayout only. The issue's second half — a CONDITIONING_AXIS_REGISTRY registering (feature_columns, encoder_module) as a pair — is deferred: it would force ConditioningConfig's fixed particle/material fields into a dynamic axis map and ripple through pipeline.py, checkpoint_io.py and rollout.py, i.e. a config-schema break with no consumer yet. - The two divergences above are unified onto build_cond_features' behaviour rather than preserved as parameters, so the new single source of truth doesn't carry the old split forward. Each gets a regression test that fails before this commit. - cat_col_layout is replaced outright (deleted, dropped from network.py's __all__, its four tests rewritten against CondLayout) rather than kept as a wrapper — two spellings of the same fact is the defect itself. cond_cat's width is now the layout's call rather than "did the caller pass a map", so an "onehot" axis without its top-N map raises instead of yielding a narrower array that ConditionEncoder would index out of bounds. pipeline.py's normalizer-fitting pass reads only cond_cont but had to be handed the maps to satisfy that. No parameter, buffer or state_dict change; existing checkpoints load unchanged, and the protected migration surfaces are untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 14:16:20 +02:00
lars	593c5f4d34	Deduplicate n_sec_head/type_head MLPs into build_mlp_head (gitea #36) ... The same two-layer classifier head (Linear(cond_out_dim, hidden_dim // 2) -> SiLU -> Linear(hidden_dim // 2, out_dim)) was hand-rolled five times in giant/model/models.py: Stage1Model.n_sec_head, Stage2OneShot.n_sec_head/ .type_head, and Stage2Autoregressive.n_sec_head/.type_head. The `// 2` ratio and fixed 2-layer depth were undocumented magic numbers, and both n_sec accuracy and secondary-species accuracy are known weak spots that were untunable independently of the trunk they hang off. Adds `build_mlp_head(in_dim, out_dim, hidden, depth, act)` to giant/model/layers.py (depth=1 is a bare Linear; depth>=2 matches the old hardcoded shape exactly), and a new `HeadConfig` (hidden_ratio, depth) dataclass in giant/config.py, wired in as `stage1_model.heads.n_sec` and `stage2_model.heads.{n_sec,type}` — split per head type (not one shared block per stage) since n_sec and species prediction are called out as separate weak spots that may want independent capacity. Defaults (hidden_ratio=0.5, depth=2) reproduce the old hardcoded architecture bit-for-bit, so every existing config.toml and migrated v0.2 checkpoint is unaffected; no changes were needed to migrate_config or the legacy migration surfaces. No new CLI flags, matching how other nested sub-config (router.*, trunk.*) is set via config.toml rather than per-field flags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 10:57:47 +02:00
lars	f301fd98d2	Make HistoryEncoder a pluggable registry, like Router/Objective (gitea #35) ... Stage2Autoregressive.init_history_cache and .history_step both isinstance-checked self.history_encoder against AttentionHistory to decide whether to use its real incremental-cache methods or a no-op fallback, so a third history type couldn't be added without editing Stage2Autoregressive itself. The two-value "markov"/"attention" enum was also independently hardcoded in three places (Stage2Autoregressive's own validation, config.py's validate_config, and AutoregressiveConfig.from_dict's default). Mirrors the Router (giant/model/routers.py) and Objective (giant/model/objectives.py, gitea #32) pattern: HistoryEncoder now declares working O(1) init_cache/step defaults (init_cache -> None, step -> one forward() call), so every registered history type satisfies the incremental interface without opting in; AttentionHistory overrides both with its real KV-cache versions since its forward() needs the full prefix. Added HISTORY_REGISTRY/register_history/build_history, registered "markov" and "attention", and deleted both isinstance checks in models.py. Per user decision during planning, config.py's validate_config now imports HISTORY_REGISTRY and checks membership dynamically instead of keeping its own hardcoded tuple, making the registry the single source of truth end to end (verified no import cycle: config.py had no prior dependency on giant.model, and giant.model.history has none on giant.config). No config-schema change and no checkpoint impact — this is a pure internal-interface refactor. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 10:35:46 +02:00
lars	f8722e347e	Add an Objective registry for the flow/ddpm/wgan generator choice (gitea #32) ... generator ∈ {"flow", "ddpm", "wgan"} was tested as a bare string in ~45 sites across models.py, sample.py, builders.py, trainers.py, and stage2_inputs.py, each independently re-deriving one of five consequences of the choice (needs a time embedding? what does the trunk take as input? is the type slice folded into the trunk output? which sampler? which loss?). giant/model/objectives.py adds an Objective ABC + OBJECTIVE_REGISTRY + build_objective factory, mirroring routers.py's Router pattern, and every bare-string site now goes through it (needs_time, is_adversarial, folds_type_slice, trunk_in_dim, build_schedule, stage1_loss/stage2_loss). Per discussion: FlowDDPMStageTrainer and WGANStageTrainer stay separate classes rather than merging into one StageTrainer as the issue's sketch proposed — their training loops are genuinely different shapes (single loss vs. dual G/D step with gradient penalty/n_critic/ST-Gumbel), and trainers.py is the least-covered-by-fast-tests part of the codebase, so a full merge was judged out of proportion to this issue's risk budget. FlowDDPMStageTrainer's own loss dispatch (flow vs ddpm, one-shot vs AR) does move onto the objective, so a future non-adversarial objective (rectified flow, consistency distillation) is still a one-file, zero-trainer-edits addition. No config-schema change — stage{1,2}_model.generator stays the persisted string, just looked up in the registry instead of string-compared. An unrecognized generator value now fails fast with a clear ValueError instead of silently falling through some bare-string checks and not others (same behavior build_router/build_trunk already have for their own type keys). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 10:10:58 +02:00
lars	0f95e0eaae	Make ResBlock's conditioning-injection mechanism selectable (gitea #34) ... ResBlock injected conditioning exactly one way — h = linear1(h) + cond_proj(cond), a conditional bias, the weakest standard option for a model whose entire job is to be conditional. Adds BLOCK_REGISTRY (giant/model/layers.py), mirroring the TRUNK_REGISTRY/ROUTER_REGISTRY registry+factory idiom (gitea #33), with two new drop-in alternatives: FilmResBlock (per-channel scale+shift modulating the norm output, zero-init so conditioning has no effect at construction) and AdaLNResBlock (DiT-style AdaLN-Zero — the norm's own affine is replaced by a conditioning-derived scale/shift, plus a zero-init gate on the residual branch, making the block the exact identity function at init). Selected per stage via a new stage{1,2}_model.trunk.block_conditioning config leaf ("add" | "film" | "adaln", default "add"), threaded through build_trunk/build_expert_body/RoutedTrunk and the three stage model constructors. Default stays "add" and ResBlock's body is unchanged, so existing configs/checkpoints are bit-identical to before this change. Decided during planning: the new field lives on the existing TrunkConfig rather than a new top-level block/blocks config section; the WGAN CriticModel (which builds its own ResBlock stack outside TRUNK_REGISTRY) and the issue's mentioned blocks.norm/blocks.activation axes are both left out of scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 09:41:50 +02:00
lars	f3f7645bf7	Make trunk architecture selectable via a registry (gitea #33) ... build_trunk hardcoded exactly two shapes (MonolithicTrunk/RoutedTrunk), chosen only by whether a Router was built, with no way to select a different trunk body architecture at all. Deviates from the issue's literal proposal (a TRUNK_REGISTRY choosing between "resmlp"/"moe" trunk shapes): during planning, decided that the trunk *body* architecture and whether it's *mixed* are orthogonal, so the registry (TRUNK_REGISTRY/register_trunk/build_expert_body in giant/model/trunks.py) holds expert bodies only (today: "resmlp", ExpertTrunk's existing input_proj -> ResBlock stack -> out_proj). Routing stays exactly router.enabled/n_experts, untouched — a future transformer body gets a mixture variant for free (trunk.type = "transformer" + router.enabled = true) instead of needing a separate registry entry per (body x routed/not) combination. MonolithicTrunk is deleted; the unrouted case now returns the registry-selected body directly, preserving today's exact state-dict keys (trunk.input_proj.* etc., not trunk.experts.0.*) — required both for existing non-routed checkpoints and because _legacy.py's migrate_legacy_state_dict already assumes that flat layout for a v0.2 checkpoint. New config leaf only: stage{1,2}_model.trunk.type: str = "resmlp" (TrunkConfig). hidden_dim/n_res_blocks/dropout stay where they are today. Nothing about router.enabled, config.migrate_config, _legacy.py, or the CLI's --router flags changes — a v0.2-migrated config gets trunk.type = "resmlp" automatically, reproducing current behaviour exactly. No CLI flag added (matches the config.toml-only precedent set by autoregressive.history/particle_type.target/n_sec.mode). No transformer body and no "none"/"linear" body (gitea #45) in this change. Full design rationale recorded on gitea #33 and #45 before implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 09:16:55 +02:00
lars	f505fe7f22	Skip router auxiliary loss compute when their lambda is 0 (gitea #31) ... FlowDDPMStageTrainer._compute unconditionally called router.balance_loss/classify_loss/entropy_loss whenever a router existed, then only added each term into total if its lambda was > 0 -- so every routed run paid for balance_loss/entropy_loss's extra router.gate(...) forward passes even at the default lambda_balance = lambda_proc = lambda_entropy = 0.0 (the exact config the failed 2026-07-22 router benchmark ran). Guard each computation on the same > 0 condition that already guarded the addition, matching WGANStageTrainer's cost structure which has no router-loss block at all. total's value is unchanged either way. Added a test that spies on the router's three loss methods and checks call counts both at lambda=0 (must be skipped) and lambda>0 (must still run, so the guard doesn't suppress the real path). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 16:18:19 +02:00
lars	32aa5a5f92	Decouple secondary-species vocabulary from conditioning.particle.emb_dim (gitea #29) ... conditioning.particle.emb_dim and stage2_model.particle_type.target="onehot"'s class count were silently the same number everywhere (pipeline.py's PDG top-N map build, Stage2OneShot/Stage2Autoregressive's type head, StageSpec's training loss width, the checkpoint's shared pdg_topn_map), fixing the secondary-species vocabulary at whatever width the unrelated physical-conditioning MLP happened to use — the exact vocabulary the v0.3.0 pivot exists to fix. Adds stage2_model.particle_type.n_classes (default 0 = inherit conditioning.particle.emb_dim, preserving today's behavior and every existing checkpoint) and a single resolve_type_n_classes helper used everywhere the coupling used to be implicit. Splits the checkpoint's shared pdg_topn_map into a conditioning-only pdg_topn_map and a new sec_type_topn_map, built independently through the existing (axis, n_classes)-keyed setup cache (no extra scan when they still resolve to the same N) and threaded through giant predict/giant rollout's decode path. A checkpoint with no sec_type_topn_map key (pre-#29) falls back to reusing pdg_topn_map, reproducing the old shared behavior exactly. Decided with the user during planning: commit directly on this branch; represent the split as an additive sec_type_topn_map checkpoint key rather than conditionally reusing pdg_topn_map; build the two top-N maps independently rather than the issue's proposed build-at-max-and-slice, since the setup cache already avoids redundant scans across runs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 16:11:14 +02:00
lars	899ca3a7d5	Validate stage2_model.autoregressive.order in validate_config (gitea #30) ... order was documented as single-valued ("energy_desc" only, placeholder for a future alternative ordering) but validate_config only checked its siblings history/teacher_forcing, so e.g. order = "energy_asc" was silently accepted and trained as if it were energy_desc. Add the missing check alongside the other two, gated the same way (only meaningful under stage2_model.decoder = "autoregressive"). Also updates the stale reason string on the pre-existing _KNOWN_UNUSED allow-list entry for this key in tests/test_config_consumed_keys.py, since half of it ("validate_config ... never [checks] order") is no longer true after this fix — the key stays allow-listed because validate_config itself isn't in that test's build/train/rollout consumer whitelist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 15:40:47 +02:00
lars	da717971b6	Honour wgan.critic_hidden_dim/critic_n_res_blocks in build_critics (gitea #28) ... build_critics always sized a WGAN critic off the generator's own hidden_dim/n_res_blocks, silently discarding the documented 0=inherit sentinel on stage{1,2}_model.wgan.critic_hidden_dim/critic_n_res_blocks (the same convention critic_lr already honoured). Now both keys are read with the 0 -> inherit fallback, and stage-scoped-only CLI flags (--stage{1,2}-critic-hidden-dim/--stage{1,2}-critic-n-res-blocks) are added -- no shared alias, since critic sizing is an architectural per-stage knob like --hidden-dim/--n-res-blocks, not a shared training hyperparameter like --n-critic/--gp-weight/--noise-dim/--critic-lr. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 15:36:26 +02:00
lars	c3fc768b40	Reject stage2_model.stage1_context = 'sampled' as unimplemented (issues.md Issue 1) ... trainers.py unconditionally trains stage 2 against the ground-truth stage-1 output (stage1_ctx = x1_s1.detach()), but 'sampled' was accepted by validate_config, stored in config.toml and the checkpoint's model_config, and silently trained identically to 'truth' — mislabeling every downstream artifact for a run launched with --stage2-stage1-context sampled. Mirrors the existing stop_token validate_config pattern. User chose the immediate fix (reject loudly) over the proper fix (actually implement sampled context), which is scoped to Issue 16. Also updates the _KNOWN_UNUSED reason for stage2_model.stage1_context (added by Issue 5's consumed-keys audit) to reflect that the value is now rejected rather than silently accepted, and drops the now-invalid --stage2-stage1-context sampled case from test_stage2_only_knobs (a full CLI invocation) — that flag's plumbing is still covered at the overrides-dict level by test_overrides_from_flags_stage2_only_knobs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 14:57:21 +02:00
lars	a4b5a6c3bf	Add consumed-keys audit test (issues.md Issue 5) ... validate_config_keys only checks that a config key is declared in DEFAULT_CONFIG, never that anything reads it — the gap that let Issues 1, 2 and 4's dead keys (stage1_context, wgan.critic_hidden_dim/critic_n_res_blocks, autoregressive.order) slip through silently. tests/test_config_consumed_keys.py walks every DEFAULT_CONFIG leaf path and asserts each is either found (via AST scan for attribute access, dict-key-shaped string constants, or constructor/ function parameter names — the last needed because Router subclasses receive their config via **kwargs filtered by signature) in a fixed whitelist of build/train/rollout consumer files, or explicitly recorded in _KNOWN_UNUSED with a reason. A second test asserts the allow-list has no stale entries, so fixing Issue 1/2/4 will force removal of the corresponding allow-list line rather than let it silently outlive the bug. The whitelist is intentionally narrower than "anywhere in giant/": scanning the whole package produces false negatives from unrelated identifier collisions (e.g. router_gating.py's unrelated `order` parameter would make autoregressive.order read as consumed). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 14:42:47 +02:00
lars	81eb14d75c	Move scripts/ to giant/tools/ (issues.md Issue 9) ... `scripts` was published as a top-level distribution package, colliding with one of the most generic names in the Python ecosystem and shadowable by a stray scripts/ dir on the portal machines' shared /work/lbogner. Move it under the giant namespace; the dwarf command name is unchanged, only the Python import path and file location move. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:31:47 +02:00
lars	a4f4cba58b	Type the data/model/training batch contracts with NamedTuples (issues.md Issue 7) ... build_features (transforms.py) now returns StepFeatures and StreamingStepsDataset (dataset.py) now yields StepBatch, both NamedTuples with the same field order as the tuples they replace, so ty can catch a dropped/added field at every consuming call site instead of a silent positional-tuple mismatch. Converted the unreadable throwaway-heavy unpacks in cli.py, pipeline.py, validate.py, and dataset.py to named attribute access; gave the WGAN path's derived 5-element batch its own _Stage2RealFakeBatch NamedTuple; updated the two test batch-construction helpers to build real StepBatchs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:11:58 +02:00
lars	e6261cea03	Unify the two v0.2->v0.3 migration surfaces (issues.md Issue 6) ... giant/config.py:migrate_config (config.toml) and giant/model/network.py:_migrate_legacy_model_config (checkpoint model_config) independently hand-maintained the same v0.2 facts and an identical router expert-sizing rejection. Extract the shared knowledge into a new leaf module, giant/_migration.py (V02_MODEL_KEY_TO_STAGES, V02_FIXED_FACTS, reject_legacy_router_expert_sizing), consumed by both. Also replace NSecConfig's legacy-only, nullable legacy_owner sentinel (living in an extra: dict catch-all) with a normal, always-set owner: str = "stage2" field, so build_models reads one concrete two-valued key instead of branching on a legacy marker. Record in CLAUDE.md that v0.2 checkpoint-loading support has no expiry decided yet, since /ceph still holds pre-v0.3.0 checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 09:56:22 +02:00
lars	818c380fd0	Extract predict/rollout's duplicated inference bootstrap into giant.checkpoint_io (issues.md Issue 5) ... giant predict and giant rollout each carried a ~65-line, independently drifting copy of "load checkpoint -> validate -> resolve conditioning axes -> restore normalizers/vocab maps -> build models -> load weights", plus a third partial copy of _conditioning_axes in analysis/router_gating.py. A silent divergence there doesn't crash, it makes the two commands run different physics from the same checkpoint with no test coverage anywhere along that path. giant/checkpoint_io.py now holds the single implementation: load_for_inference() + an InferenceContext dataclass, raising CheckpointCompatibilityError (verbatim message text preserved) instead of calling typer directly, so it can be unit-tested and imported from non-Typer code. router_gating.py's load_router imports conditioning_axes from it lazily, keeping its "no torch at module scope" contract intact. Adds 17 direct unit tests for load_for_inference/conditioning_axes/stage_cfg plus CLI smoke tests confirming the error surfaces as typer.Exit(1) through predict and rollout — previously zero coverage on this path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:31:14 +02:00
lars	2bfb1ab056	Extract giant train/new-run's CLI override mapping into a table-driven function (issues.md Issues 3 & 4) ... train()'s ~140-line hand-written flag->config translation (three different ad hoc "more specific flag wins" patterns) and new_run()'s near-verbatim copy are replaced by a shared FlagSpec/FLAG_SPECS table and overrides_from_flags() in config.py, reused by both commands. This makes the override/precedence logic directly unit-testable without CliRunner, closing coverage gaps that had zero tests (e.g. --emb-dim/--conditioning dual-axis fan-out, three of four WGAN knob legs, --stage2-generator overriding --mode, router's stage1-only asymmetry). No CLI flags, help text, or precedence semantics changed -- `giant train --help`/`giant new-run --help` are byte-identical before and after, and all previously-passing CliRunner tests still pass unmodified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:04:49 +02:00
lars	01acbfed61	Add unknown-key validation to config.toml merge (issues.md Issue 2) ... A typo like `n_res_block` for `n_res_blocks` previously merged cleanly, passed validate_config, and silently trained a model that didn't match config.toml's documented settings. merge_cli_overrides now rejects any key not present in DEFAULT_CONFIG's schema via validate_config_keys, with a did-you-mean suggestion, while still allowing the genuinely dynamic composed-router axis keys and centers_init. Checkpoint model_config loading is untouched, so old checkpoints keep loading. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 14:47:29 +02:00
lars	9bf5874308	Make config dataclasses the single source of truth for DEFAULT_CONFIG ... DEFAULT_CONFIG and build_models/build_critics/StageSpec.from_config's inline .get(key, default) fallbacks had already drifted: two keys (stage2_model.decoder, stage2_model.particle_type.target) resolved differently depending on whether a config dict came from merge_cli_overrides (fully populated, correct) or was hand-built and partial (fell back to stale v0.2-shaped literals). Introduce frozen dataclasses (GiantConfig and its nested blocks) in giant/config.py as the actual single declaration of every default; DEFAULT_CONFIG is now generated from them instead of hand-maintained, and build_models, build_critics, and StageSpec.from_config consume the dataclasses instead of duplicating literal fallbacks, so this class of drift can't recur. Router/n_sec sub-blocks keep an `extra` catch-all for their genuinely dynamic keys (composed-router axes, runtime-seeded centers_init, legacy_owner). Fixing the fallback surfaced the same latent bug in two existing partial-config callers that had been silently depending on it: a test fixture in test_train.py and scripts/warm_setup_cache.py's minimal cfg (now merged against DEFAULT_CONFIG instead of hand-rolled, closing the gap for good). See issues.md Issue 1. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 14:35:14 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	9ce7b32324	Fix test_render_all_run_gallery_invokes_subprocess clobbering LaTeX's own subprocess.run ... render_mod.subprocess is the stdlib subprocess module itself, not a copy — patching .run unconditionally also intercepted the real subprocess.run calls matplotlib's texmanager makes to compile LaTeX during savefig, so those returned the test's fake return value instead of a real CompletedProcess and crashed with AttributeError: 'NoneType' object has no attribute 'stdout' on any environment where render_all runs before the gallery call (i.e. everywhere but this dev machine's warm state that happened to mask it). Only intercept the "gallery generate" call now; everything else passes through to the real subprocess.run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:56:36 +02:00
lars	82772e4e09	Add render.py coverage: figure params, router diagnostics plots, gallery/condor glue ... render.py was at 58% coverage — the module's plotting dispatch (router_gating, router_share, unavailable) and glue logic (_figure_params/_figure_params_v2, _plot_metadata, render_all's gallery subprocess call, render_run's condor RunMeta wiring) had no tests at all. Brings it to 100%: pure-function unit tests for the v0.2/v0.3.0 figure-param branches and _plot_metadata, real LaTeX-rendered fixtures for the previously-untested plot kinds and a 4-group grouped_hist (exercises the hidden-leftover-axis branch), and mocked subprocess/condor calls to isolate render_all/render_run's own logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:47:31 +02:00
lars	24445b7427	Add coverage for router-center seeding, geometry batch reader, material topN cache, and setup-cache corruption paths ... Closes the highest-value coverage gaps found via pytest-cov: pipeline.py's EnergyRouter quantile-seeding (the roadmap's flagged fix for the failed MoE rollout benchmark) had zero coverage, geometry.py's real parquet-batch reader was always mocked, the material top-N-map cache-hit branch was untested (only pdg's was), and setup_cache.py was missing malformed-cache-body and unknown-axis error paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:38:04 +02:00
lars	878e9ddca3	Delete docs/v0.3.0-design.md and strip all references to it ... The design doc and its followups doc are no longer needed as a live reference now that the v0.3.0 redesign is implemented — comments and docstrings across the codebase cited it extensively (file path, "design doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design rationale. Removed docs/ and edited every citing comment/docstring to drop the now-dangling reference while keeping the substantive explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its trailing pointer to the deleted file. Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or "§N.N" references (repo-wide grep); ruff and ty clean; full test suite on the heaviest-touched modules (network, sample, rollout, migration, config, train) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:19:02 +02:00
lars	d3271bc798	Silence the fork-safety warning from num_workers>0 pipeline tests ... The two DataLoader-num_workers quota tests are the only ones in the file that leave num_workers>0, so they're the only ones that actually spawn forked worker subprocesses under pytest's multi-threaded process and hit Python's fork-safety DeprecationWarning. The thing under test is just the pre-flight quota-check message, emitted before the DataLoader is built. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 09:48:58 +02:00
lars	8019a80563	Refactor train.py into giant/training/ around a metrics collector ... Every metric name used to exist in four places: the dict keys each StageTrainer returned, the hardcoded _metrics_fields() column list, the ~110-line metrics_row assembly in train(), and the tqdm/summary formatting. The two had to be kept in exact correspondence by hand or csv.DictWriter would raise. Each metric is now declared once, as a MetricSpec on the trainer that computes it. MetricsCollector derives the CSV header and W&B payload from those declarations and owns all accumulation, so train() no longer carries a running sum, and every isinstance(tr, WGANStageTrainer) branch is gone — replaced by four trainer hooks (batch_loss, summary, val_objective, supports_val_loss). giant/train.py (1875 lines) becomes giant/training/: trainers.py StageSpec + shared StageTrainer base + the two subclasses metrics.py MetricSpec, MetricsCollector stage2_inputs.py the pure AR/teacher-forcing tensor helpers, moved verbatim loop.py train() (225 lines, was ~514) + graceful shutdown checkpoint.py build/load, lifted out of train()'s closures The trainers shared ~15 identical constructor arguments and copy-pasted their cosine-warmup lambda, EMA setup, state_dict/load_state_dict, resume_lr and train_mode/eval_mode. StageSpec resolves one stage's config once (constructors go from 24 and 22 keyword arguments to (spec, model, device)), the base class holds the rest, and build_stage_trainers drops from ~100 lines to 15. Metric columns are renamed to a uniform stage/split/metric scheme (stage1/train/loss, stage2/train/d_loss, stage1/lr, stage1/router/entropy, val/loss, ...). Old metrics.csv files and W&B history are not comparable. The checkpoint format is unchanged. BEHAVIOR CHANGE — WGAN best-checkpoint selection. The old code meant to score a WGAN stage on its marginal KL, but the guard `{n: kl for n in wgan_names if n not in val_loss_per_stage}` could never fire: val_loss_per_stage was pre-seeded with 0.0 for every stage, so a WGAN stage contributed a flat 0.0 and the KL was written to metrics.csv without ever influencing best.pt. val_objective now returns it as intended. On the test harness's default flow+wgan config val_loss went from 2.182 (stage 1 only) to 15.137 (stage 1 + KL 12.954), and which epoch won changed. Runs before this commit picked their best checkpoint on the non-adversarial stages alone. Written up in docs/v0.3.0-followups.md. Verified: 699 tests pass; ruff, ruff format and ty clean. Baseline-vs- refactor metrics.csv compared across five configs (flow+wgan, AR+onehot, routed, both-flow, AR-flow) — every comparable value bit-identical except val/loss where the fix applies. Resume appends without a duplicate header and reproduces a HEAD worktree's per-epoch losses and LRs exactly across the resume boundary. A refactored last.pt loads through cli.py:_load_model_weights in both raw and ema modes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-07 17:03:20 +02:00
lars	da7cde3ef9	v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies ... Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 16:12:58 +02:00
lars	200c6d243b	v0.3.0 step 7: AttentionHistory (KV-cached) + scheduled/never teacher forcing ... AttentionHistory (giant/model/network.py) adds causal self-attention over the emitted-secondary prefix as the alternative to MarkovHistory, with a parallel forward() for training and an init_cache()/step() KV-cache path for sample.py's per-slot AR inference loop, wired into Stage2Autoregressive via history="attention". giant/train.py adds _stage2_tf_prob and _assemble_stage2_ar_inputs_scheduled, mixing ground-truth history with a detached sample_secondaries_ar self-sample per slot so teacher_forcing="scheduled"/"never" close the train/inference gap teacher_forcing="always" always avoided; wired into both stage-2 AR trainers. config.py's validate_config no longer rejects these two previously unimplemented schema values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 13:15:56 +02:00
lars	93b19911f8	v0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode ... - giant/sample.py: fix every sampler's call convention against Stage1Model/Stage2OneShot's actual forward signatures (was still calling model(x, t, cond_cont, cond_cat) positionally); add sample_secondaries_ar (free-running AR loop, unsnapped history feature) and sample_stage1/sample_stage2/resolve_n_sec dispatch helpers that read each stage's generator_kind/decoder off the model instance itself. - giant/particles.py: decode_topn_class (argmax + other_policy) and decode_embedding_nearest (L1-snap + distance) turn a secondary's "onehot"/"embedding" type prediction into a concrete PDG. - giant/rollout.py: decode_secondary_identity routes all three particle_type.target values to real mass/charge; per-stage generator dispatch (drops the single shared `mode` string, adds ddpm support); L1DistCollector accumulates the §11.3 embedding-distance diagnostic. - giant/cli.py: drop the onehot/embedding-target rejection gate (narrowed to the still-unimplemented conditioning.particle/material.type=onehot axis); fix the dead model_cfg.get("mode") bug in predict/rollout. - giant/analysis/: new type_embedding_l1_distance PlotSpec, wired through the rollout YAML sidecar (no live-model call needed, unlike router_gating -- the histogram is already pre-aggregated at rollout time). - Un-xfail every test that was blocked on this step (test_rollout.py, test_flow.py, test_wgan.py, test_phase2.py, test_router.py, test_validate.py); add test_sample.py, test_type_embedding_distance.py. Known follow-up: giant/validate.py still unpacks the training val-batch as a stale 6-tuple and doesn't use the new per-stage dispatch, so marginal validation during training degrades gracefully with a warning rather than working -- not in this step's scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 10:37:57 +02:00
lars	c9d255b1c5	v0.3.0 step 5: Stage2Autoregressive (history=markov) + §11.4 grad instrumentation ... Replaces the Stage2Autoregressive stub with a real per-token secondary decoder: MarkovHistory summarizes the previous secondary, remaining-energy fraction and slot index round out the per-token conditioning, and the existing Trunk/MonolithicTrunk/RoutedTrunk machinery is reused unchanged by batching all K_MAX tokens together under teacher forcing (one parallel pass, no new trunk code). build_models/build_critics wire it in; the WGAN critic stays whole-sequence, so build_critics needs no AR-specific path. train.py's FlowDDPMStageTrainer/WGANStageTrainer gain a decoder branch, sharing optimizer/EMA/checkpoint machinery with the one-shot path. _assemble_stage2_real is now defined in terms of the new unflattened _assemble_stage2_ar_target helper, removing a near-duplicate branch. Also lands the §11.4 differentiability validation-obligation instrumentation (trunk-gradient norm from the particle-type slice vs. the continuous slices, for generator=wgan + particle_type.target=onehot) via backward hooks in _relax_onehot_type_slice, decoder-agnostic and surfaced as two new metrics.csv columns. This also fixes the standing regression where any config not explicitly overriding decoder="one_shot" crashed at build_models, since stage2_model.decoder defaults to "autoregressive" — confirmed by removing tests/test_pipeline.py's now-stale override so the default config runs end-to-end against real synthetic data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 09:36:49 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	9112e845e0	v0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite ... Replaces train.py's single global training loop with a StageTrainer hierarchy (FlowDDPMStageTrainer, WGANStageTrainer) — one per active stage, each owning its own optimizer/LR schedule/EMA and reading only the shared batch tuple (stage 2 always teacher-forces on the ground-truth x1_s1, so stages never need each other's output at train time). Supports every stage1/stage2 generator combination, including the design doc's headline mixed case (stage1=flow + stage2=wgan) and its reverse, plus stage1-only/stage2-only ablation runs, routed+gumbel stages, and checkpoint save/resume. metrics.csv/wandb logging are stage-prefixed. validate_marginals calls are guarded with a one-time warning and a Wasserstein-magnitude fallback for wgan best-checkpoint selection, since giant/sample.py still assumes stage1 always owns n_sec_head (decision 1 moved it to stage 2 by default) — deferred to design doc step 6, not silently papered over. pipeline.py's run_setup_stage/run_train_job now read the new nested config directly; the dangling resolve_expert_dims call and the --mode wgan --router rejection are both gone (routed WGAN works). cli.py's train/new-run build correctly-shaped config overrides (architecture flags -> stage1_model only per the approved decision; --mode/--n-critic/--gp-weight/--critic-lr broadcast to both stages, matching migrate_config's own precedent and avoiding a regression on the common --mode case); predict/rollout's dangling build_models tuple-unpack is fixed; new-run now tags config_version, fixing a bug where a re-loaded v0.3 config.toml would have been silently corrupted by migrate_config mistaking it for v0.2. config.py's validate_config rejects mixed particle/material conditioning types for now (ConditionEncoder supports it, the data pipeline in giant/data/transforms.py doesn't yet). analysis/render.py and router_gating.py handle both the new nested model_config shape and legacy flat checkpoints. scripts/warm_setup_cache.py updated for run_setup_stage's new signature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 11:31:49 +02:00
lars	9ce55e5013	v0.3.0 step 2: network.py refactor to composable stage models ... Decomposes the ten permutation classes in giant/model/network.py into the reusable parts from docs/v0.3.0-design.md §5: ConditionEncoder (now independently configurable per particle/material axis), ContextAdapter, Trunk/MonolithicTrunk/RoutedTrunk/ExpertTrunk, and the stage classes Stage1Model/Stage2OneShot/CriticModel (Stage2Autoregressive stubbed, raises NotImplementedError until step 4/5). build_models/build_critics now return a dict keyed by stage and accept the new nested config shape, with routed WGAN reachable for the first time (the old --mode wgan --router rejection is gone) and stage2_model.router.tie_to_stage1 sharing a literal Router instance. A v0.2 checkpoint's flat model_config auto-migrates via _migrate_legacy_model_config + migrate_legacy_state_dict, preserving the n_sec_head's attachment to Stage1Model (legacy_owner="stage1", design doc §4.1). tests/test_migration_v02_v03.py proves this bit-identical against a frozen v0.2 snapshot (tests/legacy/network_v02_snapshot.py) for both flow and wgan, both conditioning modes. scripts/check_migration_v02_v03.py is the real-checkpoint counterpart for a portal machine with /ceph access. giant/model/schedule.py's flow-matching/DDPM loss helpers are updated to the new model-call convention (t as a keyword). giant/sample.py, giant/rollout.py, and giant/validate.py are not yet updated (deferred to design doc step 6) — their exercising tests are marked xfail with that reasoning rather than silently broken. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:55:29 +02:00
lars	eb6dd27406	v0.3.0 step 1: new nested config schema, v0.2 migration shim ... Replace the single global train.mode + [model] block with the four top-level blocks docs/v0.3.0-design.md specifies ([conditioning], [stage1_model], [stage2_model], [train]), so Stage 1 and Stage 2 can run independent generative objectives and Stage 2 can train standalone. - migrate_config translates old config.toml/checkpoint dicts on load, so nothing on /ceph goes dead; loudly rejects non-zero expert_hidden_dim/expert_n_blocks, which v0.3.0 no longer supports. - merge_cli_overrides/save_config generalize from one hardcoded nesting level (model.router) to arbitrary recursive depth. - default_out_dir_name candidates move to dotted paths against the new schema, with per-stage router/generator discriminators. - validate_config adds cross-block checks the per-block schema can't express (particle_type.target=embedding needs a matching conditioning mode, tie_to_stage1 needs an active stage 1, etc). - resolve_expert_dims is deleted (experts always inherit the stage's hidden_dim/n_res_blocks now) — pipeline.py/cli.py callers are left dangling on purpose, to be updated in the network.py/train.py steps that follow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:15:20 +02:00
lars	803aae364e	format: Format tests/test_condor.py according to ruff styling	2026-08-03 14:57:45 +02:00
lars	057d637080	Fix test_write_submit_requires_synced_venv for active-venv resolution ... The venv-detection change in condor.py now checks for a giant binary next to sys.executable before falling back to repo_dir/.venv/bin/giant, and the test's own pytest venv has one — so the expected FileNotFoundError never fired. Monkeypatch sys.executable to a nonexistent path so the test exercises the fallback with neither location populated.	2026-08-03 14:52:41 +02:00
lars	ca3a2a3462	Fix CLI/tooling robustness gaps and dedupe the Conditioning enum ... - run_create_manifest gains --force; it previously overwrote an existing manifest (including holdout.manifest, which check_holdout_overlap exists specifically to protect) with no warning or backup on a second run. - _git_user_name only caught OSError, not subprocess.TimeoutExpired (a SubprocessError, not an OSError) — a slow/loaded shared portal machine could crash `dwarf bump-gen`/`bump-schema` instead of degrading to by=None as intended. - `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when the requested count exceeds ~1/4 of the machine's CPUs, matching the same shared-machine etiquette check added to giant train in the previous commit. - The Conditioning enum was independently redefined in both giant/cli.py and scripts/dwarf.py; moved to a single giant.config.Conditioning both now import, removing the drift risk of a third conditioning mode being added to one but not the other. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:49:09 +02:00
lars	ad1b8e7835	Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline ... - prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:43 +02:00
lars	ad0341a9d4	Fix training-loop checkpoint/resume and WGAN bugs ... - Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of in-progress weights before exiting mid-epoch — it previously broke out of the epoch loop before reaching the checkpoint-save block, contradicting its own printed "saving a checkpoint" message and losing all progress since the last completed epoch. Checkpoint-dict construction is factored into a shared _build_checkpoint() helper used by both the mid-epoch and end-of-epoch save paths. - WGAN LR-schedule steps_per_epoch used the wrong denominator (n_critic + 1 instead of n_critic), causing the schedule to exhaust early and LR to floor to 0 before training completed. - --critic-lr override was silently dropped on WGAN --resume (only the generator optimizer's LR was made authoritative again after load_state_dict; optimizer_d's was not). - WGAN secondary gradient-penalty forced x_hat/grad to zero for fully-masked rows (n_sec == 0, common in a shower), adding a constant ~1.0 bias into the batch-mean GP term; such rows are now excluded from the mean. - run_train_job warns (never blocks) when --num-workers exceeds ~1/4 of the machine's CPUs, per this repo's shared-portal-machine etiquette (see CLAUDE.md's Compute environment section). Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:22 +02:00
lars	5b63dfd588	Fix conditioning="physical" so it can actually generalize past training vocab ... The whole point of conditioning="physical" is generalizing to a species/material outside the training menu, but two independent code paths still hard-required training-vocab membership: - giant/data/transforms.py: build_cond_features unconditionally raised KeyError on an out-of-vocab pdg/material. _vectorized_map_lookup gains a strict=False mode (dummy index instead of raising), used only under conditioning="physical" where ConditionEncoder never reads cond_cat anyway; "embedding" mode is untouched and still raises, since cond_cat IS the conditioning signal there. - giant/rollout.py: the known_pdg termination gate still killed a track on step 1 for any pdg outside pdg_map, regardless of conditioning mode. Now skipped entirely under conditioning="physical". - giant/model/network.py: PdgRouter/ProcessRouter always build their own training-vocab nn.Embedding independent of conditioning, silently reintroducing the same limitation at the routing layer. build_models now raises loudly if conditioning="physical" is paired with either router type, rather than silently building a model that can't generalize the way it claims to. This unblocks the held-out-species/material generalization experiment against the multi-material dataset (see CLAUDE.md roadmap). Each fix has a regression test, including an end-to-end rollout test seeded with a resolvable-but-out-of-vocab PDG code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:59 +02:00
lars	74343d3e48	Add data-integrity guards against silent NaN/Inf propagation and races ... - log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:29 +02:00
lars	8065df896e	Scope wandb run config to only-active hyperparameters ... Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:46:19 +02:00
lars	5eec4c250a	Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming ... Extends default_out_dir_name's non-default-field convention to the router's new gumbel combine-weight flag and its learnable-knob toggles, so gumbel sweep configs (learn_centers on/off, learn_width, learn_temperature) resolve to distinguishable checkpoint directory names instead of colliding on the same r-<type><n> token. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:42:56 +02:00
lars	b51eafcfa5	Add opt-in straight-through Gumbel-softmax combine weights to MoE router ... Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:29:20 +02:00
lars	da5f54ea1c	Add learnable per-expert width and shared temperature to EnergyRouter ... EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 15:33:52 +02:00
lars	d656cf3109	Store a quantile grid instead of a raw reservoir sample in the setup cache ... NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely to seed EnergyRouter centers via np.quantile at load time, which alone accounted for most of the setup cache sidecar's ~2MB size (float32 values round-tripped through Python floats serialize at full double precision). Only a handful of quantile levels are ever read back, so collapse the sample to a fixed 1001-point quantile grid at save time and interpolate arbitrary levels from it at use time instead — about 100x smaller with negligible (<0.001) error on the levels that matter. Bumps the cache format version since old sidecars have no such grid to fall back on.	2026-07-30 13:32:46 +02:00
lars	de5db25e3f	Add giant new-run to scaffold a config.toml + run dir ahead of training ... Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch: `new-run` resolves CLI hyperparameter overrides into a full config.toml and run dir (reusing the existing default_out_dir_name collision-avoidance and a newly factored-out router-override helper shared with `train`), so a run can be prepared and reviewed before `giant train` actually kicks off. Also brings README up to date with the model/CLI as it actually stands (physical/embedding conditioning, WGAN/MoE-router modes, giant analyze, W&B, setup-stage caching), which had drifted back to describing the Phase-1 proof-of-concept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 12:56:22 +02:00
lars	b944bba8fb	Offset event_id per file to avoid cross-file collisions ... Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 11:36:48 +02:00