16 Commits

Author	SHA1	Message	Date
lars	87e37ebe14	Add per-stage init_from/freeze (gitea #42) ... stage{1,2}_model.active = false already trains one stage alone, but the checkpoint it writes holds only that stage, so giant rollout refuses it -- the "retrain stage 2 alone against a fixed, known-good stage 1" experiment the 2026-08-03 species failure calls for wasn't runnable end to end. Adds stage{1,2}_model.init_from (a checkpoint .pt to load this stage's weights from before training) and .freeze (never update them), symmetric across both stages. Both stages stay active = true, so both get built and both land in the output checkpoint -- the frozen stage is merely initialized from disk instead of from scratch. Decisions made during planning: - Soft freeze: forward/backward still run every batch (loss/grad_norm stay meaningful, no autograd special-casing), only optimizer.step() (and, for the frozen stage, lr_sched.step()/EMA update) is skipped -- weights are byte-identical for the whole run. This is StageTrainer._step_optimizer, shared by the non-adversarial path and both halves (generator + critic) of the WGAN path, so a frozen WGAN stage's critic freezes too. - validate_config requires init_from whenever freeze = true, unless the run is a --resume (a resumed frozen stage's weights come from the resume checkpoint instead) -- freezing a randomly-initialized model is almost certainly a mistake. - CLI flags on both `giant train` and `giant new-run` (--stage{1,2}-init-from/--stage{1,2}-freeze), matching every other per-stage model knob's existing treatment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 14:23:20 +02:00
lars	48faaee79d	Implement stage2_model.stage1_context = "sampled" (gitea #41) ... Stage 2 was trained on ground-truth stage-1 outcomes but deployed on sampled ones, and in a rollout that gap compounds over every step of every track — the same train/inference gap teacher_forcing="scheduled" already closes within stage 2, just never applied at the stage boundary. "sampled" was declared in the schema but rejected loudly by validate_config as unimplemented; this lands the real implementation. Mirrors the existing scheduled-sampling precedent rather than a hard switch: new stage2_model.ctx_p_start/ctx_p_end (defaults 1.0 -> 0.0) linearly ramp P(condition on ground truth) from epoch 0 to the final epoch, so stage 2 doesn't chase a wildly moving stage-1 target early in training. Per the plan discussed with the user: the sample is drawn from stage 1's sampling_model() (EMA weights when present, matching what inference actually deploys), mixed per example via a Bernoulli draw (never blended within a row), and validation always uses the ground truth regardless of the schedule. Fixes a latent bug the same pattern would otherwise have hit: every sampler in giant/sample.py flips its model to .eval() with no restore, so sampling from the raw (non-EMA) stage-1 model mid-step now explicitly restores its .training flag afterward to avoid silently corrupting stage 1's own training mode for the rest of the epoch. validate_config now enforces stage1_context in {"truth", "sampled"}, requires both stages active for "sampled" (nothing to sample from otherwise), range-checks ctx_p_start/ctx_p_end, and rejects the ctx_p_start = ctx_p_end = 1.0 configuration as an unadvertised no-op identical to "truth". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 12:27:13 +02:00
lars	c1c4957e2f	Implement n_sec.mode = "stop_token" for the AR secondary decoder (gitea #40) ... Stage 2's autoregressive decoder still predicted multiplicity the v0.2 way: a one-shot n_sec_head classifier over conditioning alone, run before any secondary token existed, with the AR loop then always executing k_max slots and discarding the tail. This adds a real per-slot EOS mechanism instead: - Stage2Autoregressive gains a stop_head (build_stop_head=True) that predicts P(n_sec == k | prefix) at each slot, mutually exclusive with n_sec_head (n_sec.mode = "stop_token" builds no n_sec_head at all). - sample_secondaries_ar accepts n_sec_pred=None to drive generation off the stop head instead of a pre-resolved count: each row stops the first slot its stop logit fires (stage2_model.n_sec.stop_sampling = "greedy" — the default, threshold at 0 — or "sample", a Bernoulli draw), and the whole batch loop breaks once every row has stopped, so cost scales with the realized n_sec instead of a fixed k_max. Passing n_sec_pred explicitly (the scheduled-sampling self-sample path) is unchanged. - resolve_n_sec returns None for a stop-token decoder instead of raising; rollout.py/cli.py/validate.py now derive the realized count from sample_stage2's returned sec_valid (sec_valid.sum(-1)) after sampling, rather than resolving it up front — a no-op reordering under every other n_sec.mode, where sec_valid was already built from n_sec_pred. - Training: _stop_target_and_mask (giant/training/stage2_inputs.py) builds the per-slot target/mask (one slot wider than the existing token-content sec_mask, since the stop slot itself needs supervision) and StageTrainer._stop_loss trains it with masked BCE, gated on stop_head exactly like _n_sec_loss gates on n_sec_head. Wired into both the flow/ddpm trainer and the WGAN trainer (whose skip_g_step now also checks stop_head), weighted by the existing stage2_model.n_sec.lambda — the stop head replaces n_sec_head under this mode, so no new weight key. - validate_config now accepts stop_token (requires decoder="autoregressive" and n_sec.owner="stage2") instead of always rejecting it. Decisions made during planning: stop_sampling defaults to "greedy" for deterministic rollouts; the stop head reuses stage2_model.heads.n_sec's HeadConfig shape and stage2_model.n_sec.lambda's weight rather than adding new config keys, since the two heads never coexist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 10:44:14 +02:00
lars	c4b12b5e7a	Pass ConditioningAxisConfig/ParticleTypeConfig themselves instead of raw dicts (gitea #38) ... build_models/build_critics parsed model_config into frozen dataclasses (ConditioningConfig, Stage2ModelConfig, ...) but then threw the parsed sub-objects away and passed the original raw dicts (conditioning["particle"], s2_spec.particle_type.to_dict()) down into ConditionEncoder/StageModel/etc, which re-read them with their own hardcoded .get(key, default) fallbacks — each an independent copy of a fact the dataclass already stated once. Worst instance: giant/training/trainers.py:236 converted an already-parsed ParticleTypeConfig back into a dict for no reason. Threads ConditioningAxisConfig (particle_cfg/material_cfg) and ParticleTypeConfig (particle_type_cfg) as the actual dataclass instances through every signature that used to type them dict: ConditionEncoder, StageModel/CriticModel, resolve_type_n_classes/stage2_type_dim/ stage2_trunk_sec_dim, giant/model/builders.py, giant/sample.py, giant/training/stage2_inputs.py, giant/training/trainers.py (StageSpec/ StageTrainer), giant/pipeline.py, giant/rollout.py, giant/validate.py — so ty now catches a misspelled field instead of it silently falling back. No config-schema change: config.toml/checkpoint model_config keep the same nested-dict shape; only what happens after the existing X.from_dict(...) parse changes. User-confirmed scope decision: both axes (particle_cfg/material_cfg and particle_type_cfg), not just the more heavily-duplicated particle_type_cfg axis, and not stopping at the two most literal parse-then-discard round trips — matching the issue's own proposal. Preserved-default decision: StageModel's particle_type_cfg=None sentinel (hit only by direct/test construction — build_models always passes an explicit particle_type) still resolves to ParticleTypeConfig(target= "physical"), not ParticleTypeConfig()'s own target="onehot" config-file default — switching it would have silently grown an unused, gradient-less type_head on every test that constructs Stage2OneShot/Stage2Autoregressive without particle_type_cfg=, breaking their "every param has a grad" checks. New tests in tests/test_network.py: ConditionEncoder/StageModel store the exact ConditioningAxisConfig/ParticleTypeConfig instance passed in (identity, not just equality) — no internal dict round-trip — and build_models's output carries real dataclass instances end to end, not the plain dicts it produced before this fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 15:03:55 +02:00
lars	f505fe7f22	Skip router auxiliary loss compute when their lambda is 0 (gitea #31) ... FlowDDPMStageTrainer._compute unconditionally called router.balance_loss/classify_loss/entropy_loss whenever a router existed, then only added each term into total if its lambda was > 0 -- so every routed run paid for balance_loss/entropy_loss's extra router.gate(...) forward passes even at the default lambda_balance = lambda_proc = lambda_entropy = 0.0 (the exact config the failed 2026-07-22 router benchmark ran). Guard each computation on the same > 0 condition that already guarded the addition, matching WGANStageTrainer's cost structure which has no router-loss block at all. total's value is unchanged either way. Added a test that spies on the router's three loss methods and checks call counts both at lambda=0 (must be skipped) and lambda>0 (must still run, so the guard doesn't suppress the real path). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 16:18:19 +02:00
lars	a4f4cba58b	Type the data/model/training batch contracts with NamedTuples (issues.md Issue 7) ... build_features (transforms.py) now returns StepFeatures and StreamingStepsDataset (dataset.py) now yields StepBatch, both NamedTuples with the same field order as the tuples they replace, so ty can catch a dropped/added field at every consuming call site instead of a silent positional-tuple mismatch. Converted the unreadable throwaway-heavy unpacks in cli.py, pipeline.py, validate.py, and dataset.py to named attribute access; gave the WGAN path's derived 5-element batch its own _Stage2RealFakeBatch NamedTuple; updated the two test batch-construction helpers to build real StepBatchs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:11:58 +02:00
lars	9bf5874308	Make config dataclasses the single source of truth for DEFAULT_CONFIG ... DEFAULT_CONFIG and build_models/build_critics/StageSpec.from_config's inline .get(key, default) fallbacks had already drifted: two keys (stage2_model.decoder, stage2_model.particle_type.target) resolved differently depending on whether a config dict came from merge_cli_overrides (fully populated, correct) or was hand-built and partial (fell back to stale v0.2-shaped literals). Introduce frozen dataclasses (GiantConfig and its nested blocks) in giant/config.py as the actual single declaration of every default; DEFAULT_CONFIG is now generated from them instead of hand-maintained, and build_models, build_critics, and StageSpec.from_config consume the dataclasses instead of duplicating literal fallbacks, so this class of drift can't recur. Router/n_sec sub-blocks keep an `extra` catch-all for their genuinely dynamic keys (composed-router axes, runtime-seeded centers_init, legacy_owner). Fixing the fallback surfaced the same latent bug in two existing partial-config callers that had been silently depending on it: a test fixture in test_train.py and scripts/warm_setup_cache.py's minimal cfg (now merged against DEFAULT_CONFIG instead of hand-rolled, closing the gap for good). See issues.md Issue 1. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 14:35:14 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	878e9ddca3	Delete docs/v0.3.0-design.md and strip all references to it ... The design doc and its followups doc are no longer needed as a live reference now that the v0.3.0 redesign is implemented — comments and docstrings across the codebase cited it extensively (file path, "design doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design rationale. Removed docs/ and edited every citing comment/docstring to drop the now-dangling reference while keeping the substantive explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its trailing pointer to the deleted file. Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or "§N.N" references (repo-wide grep); ruff and ty clean; full test suite on the heaviest-touched modules (network, sample, rollout, migration, config, train) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:19:02 +02:00
lars	8019a80563	Refactor train.py into giant/training/ around a metrics collector ... Every metric name used to exist in four places: the dict keys each StageTrainer returned, the hardcoded _metrics_fields() column list, the ~110-line metrics_row assembly in train(), and the tqdm/summary formatting. The two had to be kept in exact correspondence by hand or csv.DictWriter would raise. Each metric is now declared once, as a MetricSpec on the trainer that computes it. MetricsCollector derives the CSV header and W&B payload from those declarations and owns all accumulation, so train() no longer carries a running sum, and every isinstance(tr, WGANStageTrainer) branch is gone — replaced by four trainer hooks (batch_loss, summary, val_objective, supports_val_loss). giant/train.py (1875 lines) becomes giant/training/: trainers.py StageSpec + shared StageTrainer base + the two subclasses metrics.py MetricSpec, MetricsCollector stage2_inputs.py the pure AR/teacher-forcing tensor helpers, moved verbatim loop.py train() (225 lines, was ~514) + graceful shutdown checkpoint.py build/load, lifted out of train()'s closures The trainers shared ~15 identical constructor arguments and copy-pasted their cosine-warmup lambda, EMA setup, state_dict/load_state_dict, resume_lr and train_mode/eval_mode. StageSpec resolves one stage's config once (constructors go from 24 and 22 keyword arguments to (spec, model, device)), the base class holds the rest, and build_stage_trainers drops from ~100 lines to 15. Metric columns are renamed to a uniform stage/split/metric scheme (stage1/train/loss, stage2/train/d_loss, stage1/lr, stage1/router/entropy, val/loss, ...). Old metrics.csv files and W&B history are not comparable. The checkpoint format is unchanged. BEHAVIOR CHANGE — WGAN best-checkpoint selection. The old code meant to score a WGAN stage on its marginal KL, but the guard `{n: kl for n in wgan_names if n not in val_loss_per_stage}` could never fire: val_loss_per_stage was pre-seeded with 0.0 for every stage, so a WGAN stage contributed a flat 0.0 and the KL was written to metrics.csv without ever influencing best.pt. val_objective now returns it as intended. On the test harness's default flow+wgan config val_loss went from 2.182 (stage 1 only) to 15.137 (stage 1 + KL 12.954), and which epoch won changed. Runs before this commit picked their best checkpoint on the non-adversarial stages alone. Written up in docs/v0.3.0-followups.md. Verified: 699 tests pass; ruff, ruff format and ty clean. Baseline-vs- refactor metrics.csv compared across five configs (flow+wgan, AR+onehot, routed, both-flow, AR-flow) — every comparable value bit-identical except val/loss where the fix applies. Resume appends without a duplicate header and reproduces a HEAD worktree's per-epoch losses and LRs exactly across the resume boundary. A refactored last.pt loads through cli.py:_load_model_weights in both raw and ema modes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-07 17:03:20 +02:00
lars	200c6d243b	v0.3.0 step 7: AttentionHistory (KV-cached) + scheduled/never teacher forcing ... AttentionHistory (giant/model/network.py) adds causal self-attention over the emitted-secondary prefix as the alternative to MarkovHistory, with a parallel forward() for training and an init_cache()/step() KV-cache path for sample.py's per-slot AR inference loop, wired into Stage2Autoregressive via history="attention". giant/train.py adds _stage2_tf_prob and _assemble_stage2_ar_inputs_scheduled, mixing ground-truth history with a detached sample_secondaries_ar self-sample per slot so teacher_forcing="scheduled"/"never" close the train/inference gap teacher_forcing="always" always avoided; wired into both stage-2 AR trainers. config.py's validate_config no longer rejects these two previously unimplemented schema values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 13:15:56 +02:00
lars	c9d255b1c5	v0.3.0 step 5: Stage2Autoregressive (history=markov) + §11.4 grad instrumentation ... Replaces the Stage2Autoregressive stub with a real per-token secondary decoder: MarkovHistory summarizes the previous secondary, remaining-energy fraction and slot index round out the per-token conditioning, and the existing Trunk/MonolithicTrunk/RoutedTrunk machinery is reused unchanged by batching all K_MAX tokens together under teacher forcing (one parallel pass, no new trunk code). build_models/build_critics wire it in; the WGAN critic stays whole-sequence, so build_critics needs no AR-specific path. train.py's FlowDDPMStageTrainer/WGANStageTrainer gain a decoder branch, sharing optimizer/EMA/checkpoint machinery with the one-shot path. _assemble_stage2_real is now defined in terms of the new unflattened _assemble_stage2_ar_target helper, removing a near-duplicate branch. Also lands the §11.4 differentiability validation-obligation instrumentation (trunk-gradient norm from the particle-type slice vs. the continuous slices, for generator=wgan + particle_type.target=onehot) via backward hooks in _relax_onehot_type_slice, decoder-agnostic and surfaced as two new metrics.csv columns. This also fixes the standing regression where any config not explicitly overriding decoder="one_shot" crashed at build_models, since stage2_model.decoder defaults to "autoregressive" — confirmed by removing tests/test_pipeline.py's now-stale override so the default config runs end-to-end against real synthetic data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 09:36:49 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	9112e845e0	v0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite ... Replaces train.py's single global training loop with a StageTrainer hierarchy (FlowDDPMStageTrainer, WGANStageTrainer) — one per active stage, each owning its own optimizer/LR schedule/EMA and reading only the shared batch tuple (stage 2 always teacher-forces on the ground-truth x1_s1, so stages never need each other's output at train time). Supports every stage1/stage2 generator combination, including the design doc's headline mixed case (stage1=flow + stage2=wgan) and its reverse, plus stage1-only/stage2-only ablation runs, routed+gumbel stages, and checkpoint save/resume. metrics.csv/wandb logging are stage-prefixed. validate_marginals calls are guarded with a one-time warning and a Wasserstein-magnitude fallback for wgan best-checkpoint selection, since giant/sample.py still assumes stage1 always owns n_sec_head (decision 1 moved it to stage 2 by default) — deferred to design doc step 6, not silently papered over. pipeline.py's run_setup_stage/run_train_job now read the new nested config directly; the dangling resolve_expert_dims call and the --mode wgan --router rejection are both gone (routed WGAN works). cli.py's train/new-run build correctly-shaped config overrides (architecture flags -> stage1_model only per the approved decision; --mode/--n-critic/--gp-weight/--critic-lr broadcast to both stages, matching migrate_config's own precedent and avoiding a regression on the common --mode case); predict/rollout's dangling build_models tuple-unpack is fixed; new-run now tags config_version, fixing a bug where a re-loaded v0.3 config.toml would have been silently corrupted by migrate_config mistaking it for v0.2. config.py's validate_config rejects mixed particle/material conditioning types for now (ConditionEncoder supports it, the data pipeline in giant/data/transforms.py doesn't yet). analysis/render.py and router_gating.py handle both the new nested model_config shape and legacy flat checkpoints. scripts/warm_setup_cache.py updated for run_setup_stage's new signature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 11:31:49 +02:00
lars	8065df896e	Scope wandb run config to only-active hyperparameters ... Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:46:19 +02:00
lars	b51eafcfa5	Add opt-in straight-through Gumbel-softmax combine weights to MoE router ... Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:29:20 +02:00