108 Commits

Author	SHA1	Message	Date
lars	c3fc768b40	Reject stage2_model.stage1_context = 'sampled' as unimplemented (issues.md Issue 1) ... trainers.py unconditionally trains stage 2 against the ground-truth stage-1 output (stage1_ctx = x1_s1.detach()), but 'sampled' was accepted by validate_config, stored in config.toml and the checkpoint's model_config, and silently trained identically to 'truth' — mislabeling every downstream artifact for a run launched with --stage2-stage1-context sampled. Mirrors the existing stop_token validate_config pattern. User chose the immediate fix (reject loudly) over the proper fix (actually implement sampled context), which is scoped to Issue 16. Also updates the _KNOWN_UNUSED reason for stage2_model.stage1_context (added by Issue 5's consumed-keys audit) to reflect that the value is now rejected rather than silently accepted, and drops the now-invalid --stage2-stage1-context sampled case from test_stage2_only_knobs (a full CLI invocation) — that flag's plumbing is still covered at the overrides-dict level by test_overrides_from_flags_stage2_only_knobs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 14:57:21 +02:00
lars	a4b5a6c3bf	Add consumed-keys audit test (issues.md Issue 5) ... validate_config_keys only checks that a config key is declared in DEFAULT_CONFIG, never that anything reads it — the gap that let Issues 1, 2 and 4's dead keys (stage1_context, wgan.critic_hidden_dim/critic_n_res_blocks, autoregressive.order) slip through silently. tests/test_config_consumed_keys.py walks every DEFAULT_CONFIG leaf path and asserts each is either found (via AST scan for attribute access, dict-key-shaped string constants, or constructor/ function parameter names — the last needed because Router subclasses receive their config via **kwargs filtered by signature) in a fixed whitelist of build/train/rollout consumer files, or explicitly recorded in _KNOWN_UNUSED with a reason. A second test asserts the allow-list has no stale entries, so fixing Issue 1/2/4 will force removal of the corresponding allow-list line rather than let it silently outlive the bug. The whitelist is intentionally narrower than "anywhere in giant/": scanning the whole package produces false negatives from unrelated identifier collisions (e.g. router_gating.py's unrelated `order` parameter would make autoregressive.order read as consumed). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 14:42:47 +02:00
lars	81eb14d75c	Move scripts/ to giant/tools/ (issues.md Issue 9) ... `scripts` was published as a top-level distribution package, colliding with one of the most generic names in the Python ecosystem and shadowable by a stray scripts/ dir on the portal machines' shared /work/lbogner. Move it under the giant namespace; the dwarf command name is unchanged, only the Python import path and file location move. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:31:47 +02:00
lars	a4f4cba58b	Type the data/model/training batch contracts with NamedTuples (issues.md Issue 7) ... build_features (transforms.py) now returns StepFeatures and StreamingStepsDataset (dataset.py) now yields StepBatch, both NamedTuples with the same field order as the tuples they replace, so ty can catch a dropped/added field at every consuming call site instead of a silent positional-tuple mismatch. Converted the unreadable throwaway-heavy unpacks in cli.py, pipeline.py, validate.py, and dataset.py to named attribute access; gave the WGAN path's derived 5-element batch its own _Stage2RealFakeBatch NamedTuple; updated the two test batch-construction helpers to build real StepBatchs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:11:58 +02:00
lars	e6261cea03	Unify the two v0.2->v0.3 migration surfaces (issues.md Issue 6) ... giant/config.py:migrate_config (config.toml) and giant/model/network.py:_migrate_legacy_model_config (checkpoint model_config) independently hand-maintained the same v0.2 facts and an identical router expert-sizing rejection. Extract the shared knowledge into a new leaf module, giant/_migration.py (V02_MODEL_KEY_TO_STAGES, V02_FIXED_FACTS, reject_legacy_router_expert_sizing), consumed by both. Also replace NSecConfig's legacy-only, nullable legacy_owner sentinel (living in an extra: dict catch-all) with a normal, always-set owner: str = "stage2" field, so build_models reads one concrete two-valued key instead of branching on a legacy marker. Record in CLAUDE.md that v0.2 checkpoint-loading support has no expiry decided yet, since /ceph still holds pre-v0.3.0 checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 09:56:22 +02:00
lars	818c380fd0	Extract predict/rollout's duplicated inference bootstrap into giant.checkpoint_io (issues.md Issue 5) ... giant predict and giant rollout each carried a ~65-line, independently drifting copy of "load checkpoint -> validate -> resolve conditioning axes -> restore normalizers/vocab maps -> build models -> load weights", plus a third partial copy of _conditioning_axes in analysis/router_gating.py. A silent divergence there doesn't crash, it makes the two commands run different physics from the same checkpoint with no test coverage anywhere along that path. giant/checkpoint_io.py now holds the single implementation: load_for_inference() + an InferenceContext dataclass, raising CheckpointCompatibilityError (verbatim message text preserved) instead of calling typer directly, so it can be unit-tested and imported from non-Typer code. router_gating.py's load_router imports conditioning_axes from it lazily, keeping its "no torch at module scope" contract intact. Adds 17 direct unit tests for load_for_inference/conditioning_axes/stage_cfg plus CLI smoke tests confirming the error surfaces as typer.Exit(1) through predict and rollout — previously zero coverage on this path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:31:14 +02:00
lars	2bfb1ab056	Extract giant train/new-run's CLI override mapping into a table-driven function (issues.md Issues 3 & 4) ... train()'s ~140-line hand-written flag->config translation (three different ad hoc "more specific flag wins" patterns) and new_run()'s near-verbatim copy are replaced by a shared FlagSpec/FLAG_SPECS table and overrides_from_flags() in config.py, reused by both commands. This makes the override/precedence logic directly unit-testable without CliRunner, closing coverage gaps that had zero tests (e.g. --emb-dim/--conditioning dual-axis fan-out, three of four WGAN knob legs, --stage2-generator overriding --mode, router's stage1-only asymmetry). No CLI flags, help text, or precedence semantics changed -- `giant train --help`/`giant new-run --help` are byte-identical before and after, and all previously-passing CliRunner tests still pass unmodified. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:04:49 +02:00
lars	01acbfed61	Add unknown-key validation to config.toml merge (issues.md Issue 2) ... A typo like `n_res_block` for `n_res_blocks` previously merged cleanly, passed validate_config, and silently trained a model that didn't match config.toml's documented settings. merge_cli_overrides now rejects any key not present in DEFAULT_CONFIG's schema via validate_config_keys, with a did-you-mean suggestion, while still allowing the genuinely dynamic composed-router axis keys and centers_init. Checkpoint model_config loading is untouched, so old checkpoints keep loading. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 14:47:29 +02:00
lars	9bf5874308	Make config dataclasses the single source of truth for DEFAULT_CONFIG ... DEFAULT_CONFIG and build_models/build_critics/StageSpec.from_config's inline .get(key, default) fallbacks had already drifted: two keys (stage2_model.decoder, stage2_model.particle_type.target) resolved differently depending on whether a config dict came from merge_cli_overrides (fully populated, correct) or was hand-built and partial (fell back to stale v0.2-shaped literals). Introduce frozen dataclasses (GiantConfig and its nested blocks) in giant/config.py as the actual single declaration of every default; DEFAULT_CONFIG is now generated from them instead of hand-maintained, and build_models, build_critics, and StageSpec.from_config consume the dataclasses instead of duplicating literal fallbacks, so this class of drift can't recur. Router/n_sec sub-blocks keep an `extra` catch-all for their genuinely dynamic keys (composed-router axes, runtime-seeded centers_init, legacy_owner). Fixing the fallback surfaced the same latent bug in two existing partial-config callers that had been silently depending on it: a test fixture in test_train.py and scripts/warm_setup_cache.py's minimal cfg (now merged against DEFAULT_CONFIG instead of hand-rolled, closing the gap for good). See issues.md Issue 1. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 14:35:14 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	9ce7b32324	Fix test_render_all_run_gallery_invokes_subprocess clobbering LaTeX's own subprocess.run ... render_mod.subprocess is the stdlib subprocess module itself, not a copy — patching .run unconditionally also intercepted the real subprocess.run calls matplotlib's texmanager makes to compile LaTeX during savefig, so those returned the test's fake return value instead of a real CompletedProcess and crashed with AttributeError: 'NoneType' object has no attribute 'stdout' on any environment where render_all runs before the gallery call (i.e. everywhere but this dev machine's warm state that happened to mask it). Only intercept the "gallery generate" call now; everything else passes through to the real subprocess.run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:56:36 +02:00
lars	82772e4e09	Add render.py coverage: figure params, router diagnostics plots, gallery/condor glue ... render.py was at 58% coverage — the module's plotting dispatch (router_gating, router_share, unavailable) and glue logic (_figure_params/_figure_params_v2, _plot_metadata, render_all's gallery subprocess call, render_run's condor RunMeta wiring) had no tests at all. Brings it to 100%: pure-function unit tests for the v0.2/v0.3.0 figure-param branches and _plot_metadata, real LaTeX-rendered fixtures for the previously-untested plot kinds and a 4-group grouped_hist (exercises the hidden-leftover-axis branch), and mocked subprocess/condor calls to isolate render_all/render_run's own logic. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:47:31 +02:00
lars	24445b7427	Add coverage for router-center seeding, geometry batch reader, material topN cache, and setup-cache corruption paths ... Closes the highest-value coverage gaps found via pytest-cov: pipeline.py's EnergyRouter quantile-seeding (the roadmap's flagged fix for the failed MoE rollout benchmark) had zero coverage, geometry.py's real parquet-batch reader was always mocked, the material top-N-map cache-hit branch was untested (only pdg's was), and setup_cache.py was missing malformed-cache-body and unknown-axis error paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:38:04 +02:00
lars	878e9ddca3	Delete docs/v0.3.0-design.md and strip all references to it ... The design doc and its followups doc are no longer needed as a live reference now that the v0.3.0 redesign is implemented — comments and docstrings across the codebase cited it extensively (file path, "design doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design rationale. Removed docs/ and edited every citing comment/docstring to drop the now-dangling reference while keeping the substantive explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its trailing pointer to the deleted file. Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or "§N.N" references (repo-wide grep); ruff and ty clean; full test suite on the heaviest-touched modules (network, sample, rollout, migration, config, train) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:19:02 +02:00
lars	d3271bc798	Silence the fork-safety warning from num_workers>0 pipeline tests ... The two DataLoader-num_workers quota tests are the only ones in the file that leave num_workers>0, so they're the only ones that actually spawn forked worker subprocesses under pytest's multi-threaded process and hit Python's fork-safety DeprecationWarning. The thing under test is just the pre-flight quota-check message, emitted before the DataLoader is built. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 09:48:58 +02:00
lars	8019a80563	Refactor train.py into giant/training/ around a metrics collector ... Every metric name used to exist in four places: the dict keys each StageTrainer returned, the hardcoded _metrics_fields() column list, the ~110-line metrics_row assembly in train(), and the tqdm/summary formatting. The two had to be kept in exact correspondence by hand or csv.DictWriter would raise. Each metric is now declared once, as a MetricSpec on the trainer that computes it. MetricsCollector derives the CSV header and W&B payload from those declarations and owns all accumulation, so train() no longer carries a running sum, and every isinstance(tr, WGANStageTrainer) branch is gone — replaced by four trainer hooks (batch_loss, summary, val_objective, supports_val_loss). giant/train.py (1875 lines) becomes giant/training/: trainers.py StageSpec + shared StageTrainer base + the two subclasses metrics.py MetricSpec, MetricsCollector stage2_inputs.py the pure AR/teacher-forcing tensor helpers, moved verbatim loop.py train() (225 lines, was ~514) + graceful shutdown checkpoint.py build/load, lifted out of train()'s closures The trainers shared ~15 identical constructor arguments and copy-pasted their cosine-warmup lambda, EMA setup, state_dict/load_state_dict, resume_lr and train_mode/eval_mode. StageSpec resolves one stage's config once (constructors go from 24 and 22 keyword arguments to (spec, model, device)), the base class holds the rest, and build_stage_trainers drops from ~100 lines to 15. Metric columns are renamed to a uniform stage/split/metric scheme (stage1/train/loss, stage2/train/d_loss, stage1/lr, stage1/router/entropy, val/loss, ...). Old metrics.csv files and W&B history are not comparable. The checkpoint format is unchanged. BEHAVIOR CHANGE — WGAN best-checkpoint selection. The old code meant to score a WGAN stage on its marginal KL, but the guard `{n: kl for n in wgan_names if n not in val_loss_per_stage}` could never fire: val_loss_per_stage was pre-seeded with 0.0 for every stage, so a WGAN stage contributed a flat 0.0 and the KL was written to metrics.csv without ever influencing best.pt. val_objective now returns it as intended. On the test harness's default flow+wgan config val_loss went from 2.182 (stage 1 only) to 15.137 (stage 1 + KL 12.954), and which epoch won changed. Runs before this commit picked their best checkpoint on the non-adversarial stages alone. Written up in docs/v0.3.0-followups.md. Verified: 699 tests pass; ruff, ruff format and ty clean. Baseline-vs- refactor metrics.csv compared across five configs (flow+wgan, AR+onehot, routed, both-flow, AR-flow) — every comparable value bit-identical except val/loss where the fix applies. Resume appends without a duplicate header and reproduces a HEAD worktree's per-epoch losses and LRs exactly across the resume boundary. A refactored last.pt loads through cli.py:_load_model_weights in both raw and ema modes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-07 17:03:20 +02:00
lars	da7cde3ef9	v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies ... Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 16:12:58 +02:00
lars	200c6d243b	v0.3.0 step 7: AttentionHistory (KV-cached) + scheduled/never teacher forcing ... AttentionHistory (giant/model/network.py) adds causal self-attention over the emitted-secondary prefix as the alternative to MarkovHistory, with a parallel forward() for training and an init_cache()/step() KV-cache path for sample.py's per-slot AR inference loop, wired into Stage2Autoregressive via history="attention". giant/train.py adds _stage2_tf_prob and _assemble_stage2_ar_inputs_scheduled, mixing ground-truth history with a detached sample_secondaries_ar self-sample per slot so teacher_forcing="scheduled"/"never" close the train/inference gap teacher_forcing="always" always avoided; wired into both stage-2 AR trainers. config.py's validate_config no longer rejects these two previously unimplemented schema values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 13:15:56 +02:00
lars	93b19911f8	v0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode ... - giant/sample.py: fix every sampler's call convention against Stage1Model/Stage2OneShot's actual forward signatures (was still calling model(x, t, cond_cont, cond_cat) positionally); add sample_secondaries_ar (free-running AR loop, unsnapped history feature) and sample_stage1/sample_stage2/resolve_n_sec dispatch helpers that read each stage's generator_kind/decoder off the model instance itself. - giant/particles.py: decode_topn_class (argmax + other_policy) and decode_embedding_nearest (L1-snap + distance) turn a secondary's "onehot"/"embedding" type prediction into a concrete PDG. - giant/rollout.py: decode_secondary_identity routes all three particle_type.target values to real mass/charge; per-stage generator dispatch (drops the single shared `mode` string, adds ddpm support); L1DistCollector accumulates the §11.3 embedding-distance diagnostic. - giant/cli.py: drop the onehot/embedding-target rejection gate (narrowed to the still-unimplemented conditioning.particle/material.type=onehot axis); fix the dead model_cfg.get("mode") bug in predict/rollout. - giant/analysis/: new type_embedding_l1_distance PlotSpec, wired through the rollout YAML sidecar (no live-model call needed, unlike router_gating -- the histogram is already pre-aggregated at rollout time). - Un-xfail every test that was blocked on this step (test_rollout.py, test_flow.py, test_wgan.py, test_phase2.py, test_router.py, test_validate.py); add test_sample.py, test_type_embedding_distance.py. Known follow-up: giant/validate.py still unpacks the training val-batch as a stale 6-tuple and doesn't use the new per-stage dispatch, so marginal validation during training degrades gracefully with a warning rather than working -- not in this step's scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 10:37:57 +02:00
lars	c9d255b1c5	v0.3.0 step 5: Stage2Autoregressive (history=markov) + §11.4 grad instrumentation ... Replaces the Stage2Autoregressive stub with a real per-token secondary decoder: MarkovHistory summarizes the previous secondary, remaining-energy fraction and slot index round out the per-token conditioning, and the existing Trunk/MonolithicTrunk/RoutedTrunk machinery is reused unchanged by batching all K_MAX tokens together under teacher forcing (one parallel pass, no new trunk code). build_models/build_critics wire it in; the WGAN critic stays whole-sequence, so build_critics needs no AR-specific path. train.py's FlowDDPMStageTrainer/WGANStageTrainer gain a decoder branch, sharing optimizer/EMA/checkpoint machinery with the one-shot path. _assemble_stage2_real is now defined in terms of the new unflattened _assemble_stage2_ar_target helper, removing a near-duplicate branch. Also lands the §11.4 differentiability validation-obligation instrumentation (trunk-gradient norm from the particle-type slice vs. the continuous slices, for generator=wgan + particle_type.target=onehot) via backward hooks in _relax_onehot_type_slice, decoder-agnostic and surfaced as two new metrics.csv columns. This also fixes the standing regression where any config not explicitly overriding decoder="one_shot" crashed at build_models, since stage2_model.decoder defaults to "autoregressive" — confirmed by removing tests/test_pipeline.py's now-stale override so the default config runs end-to-end against real synthetic data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 09:36:49 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	9112e845e0	v0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite ... Replaces train.py's single global training loop with a StageTrainer hierarchy (FlowDDPMStageTrainer, WGANStageTrainer) — one per active stage, each owning its own optimizer/LR schedule/EMA and reading only the shared batch tuple (stage 2 always teacher-forces on the ground-truth x1_s1, so stages never need each other's output at train time). Supports every stage1/stage2 generator combination, including the design doc's headline mixed case (stage1=flow + stage2=wgan) and its reverse, plus stage1-only/stage2-only ablation runs, routed+gumbel stages, and checkpoint save/resume. metrics.csv/wandb logging are stage-prefixed. validate_marginals calls are guarded with a one-time warning and a Wasserstein-magnitude fallback for wgan best-checkpoint selection, since giant/sample.py still assumes stage1 always owns n_sec_head (decision 1 moved it to stage 2 by default) — deferred to design doc step 6, not silently papered over. pipeline.py's run_setup_stage/run_train_job now read the new nested config directly; the dangling resolve_expert_dims call and the --mode wgan --router rejection are both gone (routed WGAN works). cli.py's train/new-run build correctly-shaped config overrides (architecture flags -> stage1_model only per the approved decision; --mode/--n-critic/--gp-weight/--critic-lr broadcast to both stages, matching migrate_config's own precedent and avoiding a regression on the common --mode case); predict/rollout's dangling build_models tuple-unpack is fixed; new-run now tags config_version, fixing a bug where a re-loaded v0.3 config.toml would have been silently corrupted by migrate_config mistaking it for v0.2. config.py's validate_config rejects mixed particle/material conditioning types for now (ConditionEncoder supports it, the data pipeline in giant/data/transforms.py doesn't yet). analysis/render.py and router_gating.py handle both the new nested model_config shape and legacy flat checkpoints. scripts/warm_setup_cache.py updated for run_setup_stage's new signature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 11:31:49 +02:00
lars	9ce55e5013	v0.3.0 step 2: network.py refactor to composable stage models ... Decomposes the ten permutation classes in giant/model/network.py into the reusable parts from docs/v0.3.0-design.md §5: ConditionEncoder (now independently configurable per particle/material axis), ContextAdapter, Trunk/MonolithicTrunk/RoutedTrunk/ExpertTrunk, and the stage classes Stage1Model/Stage2OneShot/CriticModel (Stage2Autoregressive stubbed, raises NotImplementedError until step 4/5). build_models/build_critics now return a dict keyed by stage and accept the new nested config shape, with routed WGAN reachable for the first time (the old --mode wgan --router rejection is gone) and stage2_model.router.tie_to_stage1 sharing a literal Router instance. A v0.2 checkpoint's flat model_config auto-migrates via _migrate_legacy_model_config + migrate_legacy_state_dict, preserving the n_sec_head's attachment to Stage1Model (legacy_owner="stage1", design doc §4.1). tests/test_migration_v02_v03.py proves this bit-identical against a frozen v0.2 snapshot (tests/legacy/network_v02_snapshot.py) for both flow and wgan, both conditioning modes. scripts/check_migration_v02_v03.py is the real-checkpoint counterpart for a portal machine with /ceph access. giant/model/schedule.py's flow-matching/DDPM loss helpers are updated to the new model-call convention (t as a keyword). giant/sample.py, giant/rollout.py, and giant/validate.py are not yet updated (deferred to design doc step 6) — their exercising tests are marked xfail with that reasoning rather than silently broken. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:55:29 +02:00
lars	eb6dd27406	v0.3.0 step 1: new nested config schema, v0.2 migration shim ... Replace the single global train.mode + [model] block with the four top-level blocks docs/v0.3.0-design.md specifies ([conditioning], [stage1_model], [stage2_model], [train]), so Stage 1 and Stage 2 can run independent generative objectives and Stage 2 can train standalone. - migrate_config translates old config.toml/checkpoint dicts on load, so nothing on /ceph goes dead; loudly rejects non-zero expert_hidden_dim/expert_n_blocks, which v0.3.0 no longer supports. - merge_cli_overrides/save_config generalize from one hardcoded nesting level (model.router) to arbitrary recursive depth. - default_out_dir_name candidates move to dotted paths against the new schema, with per-stage router/generator discriminators. - validate_config adds cross-block checks the per-block schema can't express (particle_type.target=embedding needs a matching conditioning mode, tie_to_stage1 needs an active stage 1, etc). - resolve_expert_dims is deleted (experts always inherit the stage's hidden_dim/n_res_blocks now) — pipeline.py/cli.py callers are left dangling on purpose, to be updated in the network.py/train.py steps that follow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:15:20 +02:00
lars	803aae364e	format: Format tests/test_condor.py according to ruff styling	2026-08-03 14:57:45 +02:00
lars	057d637080	Fix test_write_submit_requires_synced_venv for active-venv resolution ... The venv-detection change in condor.py now checks for a giant binary next to sys.executable before falling back to repo_dir/.venv/bin/giant, and the test's own pytest venv has one — so the expected FileNotFoundError never fired. Monkeypatch sys.executable to a nonexistent path so the test exercises the fallback with neither location populated.	2026-08-03 14:52:41 +02:00
lars	ca3a2a3462	Fix CLI/tooling robustness gaps and dedupe the Conditioning enum ... - run_create_manifest gains --force; it previously overwrote an existing manifest (including holdout.manifest, which check_holdout_overlap exists specifically to protect) with no warning or backup on a second run. - _git_user_name only caught OSError, not subprocess.TimeoutExpired (a SubprocessError, not an OSError) — a slow/loaded shared portal machine could crash `dwarf bump-gen`/`bump-schema` instead of degrading to by=None as intended. - `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when the requested count exceeds ~1/4 of the machine's CPUs, matching the same shared-machine etiquette check added to giant train in the previous commit. - The Conditioning enum was independently redefined in both giant/cli.py and scripts/dwarf.py; moved to a single giant.config.Conditioning both now import, removing the drift risk of a third conditioning mode being added to one but not the other. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:49:09 +02:00
lars	ad1b8e7835	Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline ... - prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:43 +02:00
lars	ad0341a9d4	Fix training-loop checkpoint/resume and WGAN bugs ... - Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of in-progress weights before exiting mid-epoch — it previously broke out of the epoch loop before reaching the checkpoint-save block, contradicting its own printed "saving a checkpoint" message and losing all progress since the last completed epoch. Checkpoint-dict construction is factored into a shared _build_checkpoint() helper used by both the mid-epoch and end-of-epoch save paths. - WGAN LR-schedule steps_per_epoch used the wrong denominator (n_critic + 1 instead of n_critic), causing the schedule to exhaust early and LR to floor to 0 before training completed. - --critic-lr override was silently dropped on WGAN --resume (only the generator optimizer's LR was made authoritative again after load_state_dict; optimizer_d's was not). - WGAN secondary gradient-penalty forced x_hat/grad to zero for fully-masked rows (n_sec == 0, common in a shower), adding a constant ~1.0 bias into the batch-mean GP term; such rows are now excluded from the mean. - run_train_job warns (never blocks) when --num-workers exceeds ~1/4 of the machine's CPUs, per this repo's shared-portal-machine etiquette (see CLAUDE.md's Compute environment section). Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:22 +02:00
lars	5b63dfd588	Fix conditioning="physical" so it can actually generalize past training vocab ... The whole point of conditioning="physical" is generalizing to a species/material outside the training menu, but two independent code paths still hard-required training-vocab membership: - giant/data/transforms.py: build_cond_features unconditionally raised KeyError on an out-of-vocab pdg/material. _vectorized_map_lookup gains a strict=False mode (dummy index instead of raising), used only under conditioning="physical" where ConditionEncoder never reads cond_cat anyway; "embedding" mode is untouched and still raises, since cond_cat IS the conditioning signal there. - giant/rollout.py: the known_pdg termination gate still killed a track on step 1 for any pdg outside pdg_map, regardless of conditioning mode. Now skipped entirely under conditioning="physical". - giant/model/network.py: PdgRouter/ProcessRouter always build their own training-vocab nn.Embedding independent of conditioning, silently reintroducing the same limitation at the routing layer. build_models now raises loudly if conditioning="physical" is paired with either router type, rather than silently building a model that can't generalize the way it claims to. This unblocks the held-out-species/material generalization experiment against the multi-material dataset (see CLAUDE.md roadmap). Each fix has a regression test, including an end-to-end rollout test seeded with a resolvable-but-out-of-vocab PDG code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:59 +02:00
lars	74343d3e48	Add data-integrity guards against silent NaN/Inf propagation and races ... - log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:29 +02:00
lars	8065df896e	Scope wandb run config to only-active hyperparameters ... Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:46:19 +02:00
lars	5eec4c250a	Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming ... Extends default_out_dir_name's non-default-field convention to the router's new gumbel combine-weight flag and its learnable-knob toggles, so gumbel sweep configs (learn_centers on/off, learn_width, learn_temperature) resolve to distinguishable checkpoint directory names instead of colliding on the same r-<type><n> token. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:42:56 +02:00
lars	b51eafcfa5	Add opt-in straight-through Gumbel-softmax combine weights to MoE router ... Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:29:20 +02:00
lars	da5f54ea1c	Add learnable per-expert width and shared temperature to EnergyRouter ... EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 15:33:52 +02:00
lars	d656cf3109	Store a quantile grid instead of a raw reservoir sample in the setup cache ... NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely to seed EnergyRouter centers via np.quantile at load time, which alone accounted for most of the setup cache sidecar's ~2MB size (float32 values round-tripped through Python floats serialize at full double precision). Only a handful of quantile levels are ever read back, so collapse the sample to a fixed 1001-point quantile grid at save time and interpolate arbitrary levels from it at use time instead — about 100x smaller with negligible (<0.001) error on the levels that matter. Bumps the cache format version since old sidecars have no such grid to fall back on.	2026-07-30 13:32:46 +02:00
lars	de5db25e3f	Add giant new-run to scaffold a config.toml + run dir ahead of training ... Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch: `new-run` resolves CLI hyperparameter overrides into a full config.toml and run dir (reusing the existing default_out_dir_name collision-avoidance and a newly factored-out router-override helper shared with `train`), so a run can be prepared and reviewed before `giant train` actually kicks off. Also brings README up to date with the model/CLI as it actually stands (physical/embedding conditioning, WGAN/MoE-router modes, giant analyze, W&B, setup-stage caching), which had drifted back to describing the Phase-1 proof-of-concept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 12:56:22 +02:00
lars	b944bba8fb	Offset event_id per file to avoid cross-file collisions ... Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 11:36:48 +02:00
lars	471a81b5e7	Add dwarf warm-cache to precompute the setup-stage sidecar ... Lets the vocab maps, event-id split index, and normalizer stats be warmed once for a dataset (right after `dwarf convert`, or before a `dwarf hparam-scan` sweep) without needing to also start training. Extracts the setup-stage logic out of giant/pipeline.py:run_train_job into a standalone run_setup_stage() (returning a SetupStageResult), reused by both run_train_job and the new dwarf command's scripts/warm_setup_cache.py — a behavior-preserving refactor, covered by the existing test_pipeline.py suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:53:31 +02:00
lars	e7478c36fb	Cache giant train's setup stage in a sidecar file ... Building the pdg/material vocab maps, the process map, and fitting the Stage-1/Stage-2 normalizers all require scanning the training dataset before a single epoch runs, which is wasted work whenever the same data path is reused across runs (hyperparameter sweeps via `dwarf hparam-scan`, repeated manual training attempts, ...). Persist those setup-stage outputs to a JSON sidecar next to the input data (giant/data/setup_cache.py), validated by a file fingerprint plus fixed dimension constants and a manually-bumped format version before reuse, with a soft warning (not a hard invalidation) on a git-hash mismatch alone. Also derives n_train_steps instantly from cached per-event row counts instead of accumulating it during the normalizer scan, and always collects the energy-router reservoir sample while the cache is being populated (not only when the current run's router is energy-typed) so a later run enabling --router-type energy never needs to rescan just to seed expert centers. New --cache-setup/--no-cache-setup (default on) and --rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:28:37 +02:00
lars	84efbf5c2c	Merge branch 'master' into perf/setup-stage	2026-07-29 13:56:59 +02:00
lars	759b67a9e1	Speed up _WelfordAccumulator's per-chunk update ... The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:55:07 +02:00
lars	47a6c9db1f	Add regression coverage for vocab/process index-map builders ... build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:45:33 +02:00
lars	09e4c765c7	Speed up giant train's setup stage ... Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:34:32 +02:00
lars	1115451c8e	Make default checkpoint out_dir name reflect only non-default hyperparams ... Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs.	2026-07-29 11:21:02 +02:00
lars	539b6f61e1	Add test coverage for resolve_expert_dims ... Covers the default-config 0-sentinel inheritance path (the exact bug fixed by 969c5c6, previously untested since every router test in test_router.py passes expert_hidden_dim/expert_n_blocks explicitly), plus missing-key inheritance, full override, and partial override. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:44 +02:00
lars	ee29b9a303	router: seed EnergyRouter centers from data quantiles instead of a fixed linspace ... The 2026-07-22 rollout benchmark's router_gating diagnostic showed the 10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a roughly uniform z-normalized energy distribution, leaving experts heavily overlapping instead of partitioning the range. Add an optional centers_init kwarg (backward compatible, defaults to the old linspace) and have giant train estimate it from a reservoir sample of the real energy column, collected during the existing normalizer-fitting pass.	2026-07-27 13:11:16 +02:00
lars	61410ddee3	analyze: default run directory to <repo>/analysis_runs, gitignored ... giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.	2026-07-27 11:29:35 +02:00
lars	eb751d968d	transforms: pad legacy cond normalizers for pre-physical-conditioning checkpoints ... Checkpoints trained before commit 68fb99b (physical-property conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond normalizer, fit before build_cond_features grew the extra physical columns. Any inference against such a checkpoint under current code (predict/rollout/router_gating) crashed broadcasting a 15-wide cond_cont against an 8-wide mean/std. In "embedding" mode those physical columns are never read by ConditionEncoder, so padding the missing entries with mean=0/std=1 is a safe no-op. "physical" mode reads them directly, so a mismatch there still raises instead of silently normalizing garbage.	2026-07-27 11:06:48 +02:00
lars	dbd5c7e083	analyze: default condor docker image to alma9-gridjob ... mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9 not found). Switch the default to cverstege/alma9-gridjob, a modern EL9-based image.	2026-07-27 10:14:15 +02:00