96 Commits

Author	SHA1	Message	Date
lars	24445b7427	Add coverage for router-center seeding, geometry batch reader, material topN cache, and setup-cache corruption paths ... Closes the highest-value coverage gaps found via pytest-cov: pipeline.py's EnergyRouter quantile-seeding (the roadmap's flagged fix for the failed MoE rollout benchmark) had zero coverage, geometry.py's real parquet-batch reader was always mocked, the material top-N-map cache-hit branch was untested (only pdg's was), and setup_cache.py was missing malformed-cache-body and unknown-axis error paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:38:04 +02:00
lars	878e9ddca3	Delete docs/v0.3.0-design.md and strip all references to it ... The design doc and its followups doc are no longer needed as a live reference now that the v0.3.0 redesign is implemented — comments and docstrings across the codebase cited it extensively (file path, "design doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design rationale. Removed docs/ and edited every citing comment/docstring to drop the now-dangling reference while keeping the substantive explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its trailing pointer to the deleted file. Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or "§N.N" references (repo-wide grep); ruff and ty clean; full test suite on the heaviest-touched modules (network, sample, rollout, migration, config, train) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:19:02 +02:00
lars	d3271bc798	Silence the fork-safety warning from num_workers>0 pipeline tests ... The two DataLoader-num_workers quota tests are the only ones in the file that leave num_workers>0, so they're the only ones that actually spawn forked worker subprocesses under pytest's multi-threaded process and hit Python's fork-safety DeprecationWarning. The thing under test is just the pre-flight quota-check message, emitted before the DataLoader is built. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 09:48:58 +02:00
lars	8019a80563	Refactor train.py into giant/training/ around a metrics collector ... Every metric name used to exist in four places: the dict keys each StageTrainer returned, the hardcoded _metrics_fields() column list, the ~110-line metrics_row assembly in train(), and the tqdm/summary formatting. The two had to be kept in exact correspondence by hand or csv.DictWriter would raise. Each metric is now declared once, as a MetricSpec on the trainer that computes it. MetricsCollector derives the CSV header and W&B payload from those declarations and owns all accumulation, so train() no longer carries a running sum, and every isinstance(tr, WGANStageTrainer) branch is gone — replaced by four trainer hooks (batch_loss, summary, val_objective, supports_val_loss). giant/train.py (1875 lines) becomes giant/training/: trainers.py StageSpec + shared StageTrainer base + the two subclasses metrics.py MetricSpec, MetricsCollector stage2_inputs.py the pure AR/teacher-forcing tensor helpers, moved verbatim loop.py train() (225 lines, was ~514) + graceful shutdown checkpoint.py build/load, lifted out of train()'s closures The trainers shared ~15 identical constructor arguments and copy-pasted their cosine-warmup lambda, EMA setup, state_dict/load_state_dict, resume_lr and train_mode/eval_mode. StageSpec resolves one stage's config once (constructors go from 24 and 22 keyword arguments to (spec, model, device)), the base class holds the rest, and build_stage_trainers drops from ~100 lines to 15. Metric columns are renamed to a uniform stage/split/metric scheme (stage1/train/loss, stage2/train/d_loss, stage1/lr, stage1/router/entropy, val/loss, ...). Old metrics.csv files and W&B history are not comparable. The checkpoint format is unchanged. BEHAVIOR CHANGE — WGAN best-checkpoint selection. The old code meant to score a WGAN stage on its marginal KL, but the guard `{n: kl for n in wgan_names if n not in val_loss_per_stage}` could never fire: val_loss_per_stage was pre-seeded with 0.0 for every stage, so a WGAN stage contributed a flat 0.0 and the KL was written to metrics.csv without ever influencing best.pt. val_objective now returns it as intended. On the test harness's default flow+wgan config val_loss went from 2.182 (stage 1 only) to 15.137 (stage 1 + KL 12.954), and which epoch won changed. Runs before this commit picked their best checkpoint on the non-adversarial stages alone. Written up in docs/v0.3.0-followups.md. Verified: 699 tests pass; ruff, ruff format and ty clean. Baseline-vs- refactor metrics.csv compared across five configs (flow+wgan, AR+onehot, routed, both-flow, AR-flow) — every comparable value bit-identical except val/loss where the fix applies. Resume appends without a duplicate header and reproduces a HEAD worktree's per-epoch losses and LRs exactly across the resume boundary. A refactored last.pt loads through cli.py:_load_model_weights in both raw and ema modes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-07 17:03:20 +02:00
lars	da7cde3ef9	v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies ... Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 16:12:58 +02:00
lars	200c6d243b	v0.3.0 step 7: AttentionHistory (KV-cached) + scheduled/never teacher forcing ... AttentionHistory (giant/model/network.py) adds causal self-attention over the emitted-secondary prefix as the alternative to MarkovHistory, with a parallel forward() for training and an init_cache()/step() KV-cache path for sample.py's per-slot AR inference loop, wired into Stage2Autoregressive via history="attention". giant/train.py adds _stage2_tf_prob and _assemble_stage2_ar_inputs_scheduled, mixing ground-truth history with a detached sample_secondaries_ar self-sample per slot so teacher_forcing="scheduled"/"never" close the train/inference gap teacher_forcing="always" always avoided; wired into both stage-2 AR trainers. config.py's validate_config no longer rejects these two previously unimplemented schema values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 13:15:56 +02:00
lars	93b19911f8	v0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode ... - giant/sample.py: fix every sampler's call convention against Stage1Model/Stage2OneShot's actual forward signatures (was still calling model(x, t, cond_cont, cond_cat) positionally); add sample_secondaries_ar (free-running AR loop, unsnapped history feature) and sample_stage1/sample_stage2/resolve_n_sec dispatch helpers that read each stage's generator_kind/decoder off the model instance itself. - giant/particles.py: decode_topn_class (argmax + other_policy) and decode_embedding_nearest (L1-snap + distance) turn a secondary's "onehot"/"embedding" type prediction into a concrete PDG. - giant/rollout.py: decode_secondary_identity routes all three particle_type.target values to real mass/charge; per-stage generator dispatch (drops the single shared `mode` string, adds ddpm support); L1DistCollector accumulates the §11.3 embedding-distance diagnostic. - giant/cli.py: drop the onehot/embedding-target rejection gate (narrowed to the still-unimplemented conditioning.particle/material.type=onehot axis); fix the dead model_cfg.get("mode") bug in predict/rollout. - giant/analysis/: new type_embedding_l1_distance PlotSpec, wired through the rollout YAML sidecar (no live-model call needed, unlike router_gating -- the histogram is already pre-aggregated at rollout time). - Un-xfail every test that was blocked on this step (test_rollout.py, test_flow.py, test_wgan.py, test_phase2.py, test_router.py, test_validate.py); add test_sample.py, test_type_embedding_distance.py. Known follow-up: giant/validate.py still unpacks the training val-batch as a stale 6-tuple and doesn't use the new per-stage dispatch, so marginal validation during training degrades gracefully with a warning rather than working -- not in this step's scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 10:37:57 +02:00
lars	c9d255b1c5	v0.3.0 step 5: Stage2Autoregressive (history=markov) + §11.4 grad instrumentation ... Replaces the Stage2Autoregressive stub with a real per-token secondary decoder: MarkovHistory summarizes the previous secondary, remaining-energy fraction and slot index round out the per-token conditioning, and the existing Trunk/MonolithicTrunk/RoutedTrunk machinery is reused unchanged by batching all K_MAX tokens together under teacher forcing (one parallel pass, no new trunk code). build_models/build_critics wire it in; the WGAN critic stays whole-sequence, so build_critics needs no AR-specific path. train.py's FlowDDPMStageTrainer/WGANStageTrainer gain a decoder branch, sharing optimizer/EMA/checkpoint machinery with the one-shot path. _assemble_stage2_real is now defined in terms of the new unflattened _assemble_stage2_ar_target helper, removing a near-duplicate branch. Also lands the §11.4 differentiability validation-obligation instrumentation (trunk-gradient norm from the particle-type slice vs. the continuous slices, for generator=wgan + particle_type.target=onehot) via backward hooks in _relax_onehot_type_slice, decoder-agnostic and surfaced as two new metrics.csv columns. This also fixes the standing regression where any config not explicitly overriding decoder="one_shot" crashed at build_models, since stage2_model.decoder defaults to "autoregressive" — confirmed by removing tests/test_pipeline.py's now-stale override so the default config runs end-to-end against real synthetic data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 09:36:49 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	9112e845e0	v0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite ... Replaces train.py's single global training loop with a StageTrainer hierarchy (FlowDDPMStageTrainer, WGANStageTrainer) — one per active stage, each owning its own optimizer/LR schedule/EMA and reading only the shared batch tuple (stage 2 always teacher-forces on the ground-truth x1_s1, so stages never need each other's output at train time). Supports every stage1/stage2 generator combination, including the design doc's headline mixed case (stage1=flow + stage2=wgan) and its reverse, plus stage1-only/stage2-only ablation runs, routed+gumbel stages, and checkpoint save/resume. metrics.csv/wandb logging are stage-prefixed. validate_marginals calls are guarded with a one-time warning and a Wasserstein-magnitude fallback for wgan best-checkpoint selection, since giant/sample.py still assumes stage1 always owns n_sec_head (decision 1 moved it to stage 2 by default) — deferred to design doc step 6, not silently papered over. pipeline.py's run_setup_stage/run_train_job now read the new nested config directly; the dangling resolve_expert_dims call and the --mode wgan --router rejection are both gone (routed WGAN works). cli.py's train/new-run build correctly-shaped config overrides (architecture flags -> stage1_model only per the approved decision; --mode/--n-critic/--gp-weight/--critic-lr broadcast to both stages, matching migrate_config's own precedent and avoiding a regression on the common --mode case); predict/rollout's dangling build_models tuple-unpack is fixed; new-run now tags config_version, fixing a bug where a re-loaded v0.3 config.toml would have been silently corrupted by migrate_config mistaking it for v0.2. config.py's validate_config rejects mixed particle/material conditioning types for now (ConditionEncoder supports it, the data pipeline in giant/data/transforms.py doesn't yet). analysis/render.py and router_gating.py handle both the new nested model_config shape and legacy flat checkpoints. scripts/warm_setup_cache.py updated for run_setup_stage's new signature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 11:31:49 +02:00
lars	9ce55e5013	v0.3.0 step 2: network.py refactor to composable stage models ... Decomposes the ten permutation classes in giant/model/network.py into the reusable parts from docs/v0.3.0-design.md §5: ConditionEncoder (now independently configurable per particle/material axis), ContextAdapter, Trunk/MonolithicTrunk/RoutedTrunk/ExpertTrunk, and the stage classes Stage1Model/Stage2OneShot/CriticModel (Stage2Autoregressive stubbed, raises NotImplementedError until step 4/5). build_models/build_critics now return a dict keyed by stage and accept the new nested config shape, with routed WGAN reachable for the first time (the old --mode wgan --router rejection is gone) and stage2_model.router.tie_to_stage1 sharing a literal Router instance. A v0.2 checkpoint's flat model_config auto-migrates via _migrate_legacy_model_config + migrate_legacy_state_dict, preserving the n_sec_head's attachment to Stage1Model (legacy_owner="stage1", design doc §4.1). tests/test_migration_v02_v03.py proves this bit-identical against a frozen v0.2 snapshot (tests/legacy/network_v02_snapshot.py) for both flow and wgan, both conditioning modes. scripts/check_migration_v02_v03.py is the real-checkpoint counterpart for a portal machine with /ceph access. giant/model/schedule.py's flow-matching/DDPM loss helpers are updated to the new model-call convention (t as a keyword). giant/sample.py, giant/rollout.py, and giant/validate.py are not yet updated (deferred to design doc step 6) — their exercising tests are marked xfail with that reasoning rather than silently broken. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:55:29 +02:00
lars	eb6dd27406	v0.3.0 step 1: new nested config schema, v0.2 migration shim ... Replace the single global train.mode + [model] block with the four top-level blocks docs/v0.3.0-design.md specifies ([conditioning], [stage1_model], [stage2_model], [train]), so Stage 1 and Stage 2 can run independent generative objectives and Stage 2 can train standalone. - migrate_config translates old config.toml/checkpoint dicts on load, so nothing on /ceph goes dead; loudly rejects non-zero expert_hidden_dim/expert_n_blocks, which v0.3.0 no longer supports. - merge_cli_overrides/save_config generalize from one hardcoded nesting level (model.router) to arbitrary recursive depth. - default_out_dir_name candidates move to dotted paths against the new schema, with per-stage router/generator discriminators. - validate_config adds cross-block checks the per-block schema can't express (particle_type.target=embedding needs a matching conditioning mode, tie_to_stage1 needs an active stage 1, etc). - resolve_expert_dims is deleted (experts always inherit the stage's hidden_dim/n_res_blocks now) — pipeline.py/cli.py callers are left dangling on purpose, to be updated in the network.py/train.py steps that follow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:15:20 +02:00
lars	803aae364e	format: Format tests/test_condor.py according to ruff styling	2026-08-03 14:57:45 +02:00
lars	057d637080	Fix test_write_submit_requires_synced_venv for active-venv resolution ... The venv-detection change in condor.py now checks for a giant binary next to sys.executable before falling back to repo_dir/.venv/bin/giant, and the test's own pytest venv has one — so the expected FileNotFoundError never fired. Monkeypatch sys.executable to a nonexistent path so the test exercises the fallback with neither location populated.	2026-08-03 14:52:41 +02:00
lars	ca3a2a3462	Fix CLI/tooling robustness gaps and dedupe the Conditioning enum ... - run_create_manifest gains --force; it previously overwrote an existing manifest (including holdout.manifest, which check_holdout_overlap exists specifically to protect) with no warning or backup on a second run. - _git_user_name only caught OSError, not subprocess.TimeoutExpired (a SubprocessError, not an OSError) — a slow/loaded shared portal machine could crash `dwarf bump-gen`/`bump-schema` instead of degrading to by=None as intended. - `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when the requested count exceeds ~1/4 of the machine's CPUs, matching the same shared-machine etiquette check added to giant train in the previous commit. - The Conditioning enum was independently redefined in both giant/cli.py and scripts/dwarf.py; moved to a single giant.config.Conditioning both now import, removing the drift risk of a third conditioning mode being added to one but not the other. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:49:09 +02:00
lars	ad1b8e7835	Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline ... - prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:43 +02:00
lars	ad0341a9d4	Fix training-loop checkpoint/resume and WGAN bugs ... - Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of in-progress weights before exiting mid-epoch — it previously broke out of the epoch loop before reaching the checkpoint-save block, contradicting its own printed "saving a checkpoint" message and losing all progress since the last completed epoch. Checkpoint-dict construction is factored into a shared _build_checkpoint() helper used by both the mid-epoch and end-of-epoch save paths. - WGAN LR-schedule steps_per_epoch used the wrong denominator (n_critic + 1 instead of n_critic), causing the schedule to exhaust early and LR to floor to 0 before training completed. - --critic-lr override was silently dropped on WGAN --resume (only the generator optimizer's LR was made authoritative again after load_state_dict; optimizer_d's was not). - WGAN secondary gradient-penalty forced x_hat/grad to zero for fully-masked rows (n_sec == 0, common in a shower), adding a constant ~1.0 bias into the batch-mean GP term; such rows are now excluded from the mean. - run_train_job warns (never blocks) when --num-workers exceeds ~1/4 of the machine's CPUs, per this repo's shared-portal-machine etiquette (see CLAUDE.md's Compute environment section). Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:22 +02:00
lars	5b63dfd588	Fix conditioning="physical" so it can actually generalize past training vocab ... The whole point of conditioning="physical" is generalizing to a species/material outside the training menu, but two independent code paths still hard-required training-vocab membership: - giant/data/transforms.py: build_cond_features unconditionally raised KeyError on an out-of-vocab pdg/material. _vectorized_map_lookup gains a strict=False mode (dummy index instead of raising), used only under conditioning="physical" where ConditionEncoder never reads cond_cat anyway; "embedding" mode is untouched and still raises, since cond_cat IS the conditioning signal there. - giant/rollout.py: the known_pdg termination gate still killed a track on step 1 for any pdg outside pdg_map, regardless of conditioning mode. Now skipped entirely under conditioning="physical". - giant/model/network.py: PdgRouter/ProcessRouter always build their own training-vocab nn.Embedding independent of conditioning, silently reintroducing the same limitation at the routing layer. build_models now raises loudly if conditioning="physical" is paired with either router type, rather than silently building a model that can't generalize the way it claims to. This unblocks the held-out-species/material generalization experiment against the multi-material dataset (see CLAUDE.md roadmap). Each fix has a regression test, including an end-to-end rollout test seeded with a resolvable-but-out-of-vocab PDG code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:59 +02:00
lars	74343d3e48	Add data-integrity guards against silent NaN/Inf propagation and races ... - log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:29 +02:00
lars	8065df896e	Scope wandb run config to only-active hyperparameters ... Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:46:19 +02:00
lars	5eec4c250a	Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming ... Extends default_out_dir_name's non-default-field convention to the router's new gumbel combine-weight flag and its learnable-knob toggles, so gumbel sweep configs (learn_centers on/off, learn_width, learn_temperature) resolve to distinguishable checkpoint directory names instead of colliding on the same r-<type><n> token. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:42:56 +02:00
lars	b51eafcfa5	Add opt-in straight-through Gumbel-softmax combine weights to MoE router ... Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:29:20 +02:00
lars	da5f54ea1c	Add learnable per-expert width and shared temperature to EnergyRouter ... EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 15:33:52 +02:00
lars	d656cf3109	Store a quantile grid instead of a raw reservoir sample in the setup cache ... NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely to seed EnergyRouter centers via np.quantile at load time, which alone accounted for most of the setup cache sidecar's ~2MB size (float32 values round-tripped through Python floats serialize at full double precision). Only a handful of quantile levels are ever read back, so collapse the sample to a fixed 1001-point quantile grid at save time and interpolate arbitrary levels from it at use time instead — about 100x smaller with negligible (<0.001) error on the levels that matter. Bumps the cache format version since old sidecars have no such grid to fall back on.	2026-07-30 13:32:46 +02:00
lars	de5db25e3f	Add giant new-run to scaffold a config.toml + run dir ahead of training ... Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch: `new-run` resolves CLI hyperparameter overrides into a full config.toml and run dir (reusing the existing default_out_dir_name collision-avoidance and a newly factored-out router-override helper shared with `train`), so a run can be prepared and reviewed before `giant train` actually kicks off. Also brings README up to date with the model/CLI as it actually stands (physical/embedding conditioning, WGAN/MoE-router modes, giant analyze, W&B, setup-stage caching), which had drifted back to describing the Phase-1 proof-of-concept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 12:56:22 +02:00
lars	b944bba8fb	Offset event_id per file to avoid cross-file collisions ... Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 11:36:48 +02:00
lars	471a81b5e7	Add dwarf warm-cache to precompute the setup-stage sidecar ... Lets the vocab maps, event-id split index, and normalizer stats be warmed once for a dataset (right after `dwarf convert`, or before a `dwarf hparam-scan` sweep) without needing to also start training. Extracts the setup-stage logic out of giant/pipeline.py:run_train_job into a standalone run_setup_stage() (returning a SetupStageResult), reused by both run_train_job and the new dwarf command's scripts/warm_setup_cache.py — a behavior-preserving refactor, covered by the existing test_pipeline.py suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:53:31 +02:00
lars	e7478c36fb	Cache giant train's setup stage in a sidecar file ... Building the pdg/material vocab maps, the process map, and fitting the Stage-1/Stage-2 normalizers all require scanning the training dataset before a single epoch runs, which is wasted work whenever the same data path is reused across runs (hyperparameter sweeps via `dwarf hparam-scan`, repeated manual training attempts, ...). Persist those setup-stage outputs to a JSON sidecar next to the input data (giant/data/setup_cache.py), validated by a file fingerprint plus fixed dimension constants and a manually-bumped format version before reuse, with a soft warning (not a hard invalidation) on a git-hash mismatch alone. Also derives n_train_steps instantly from cached per-event row counts instead of accumulating it during the normalizer scan, and always collects the energy-router reservoir sample while the cache is being populated (not only when the current run's router is energy-typed) so a later run enabling --router-type energy never needs to rescan just to seed expert centers. New --cache-setup/--no-cache-setup (default on) and --rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:28:37 +02:00
lars	84efbf5c2c	Merge branch 'master' into perf/setup-stage	2026-07-29 13:56:59 +02:00
lars	759b67a9e1	Speed up _WelfordAccumulator's per-chunk update ... The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:55:07 +02:00
lars	47a6c9db1f	Add regression coverage for vocab/process index-map builders ... build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:45:33 +02:00
lars	09e4c765c7	Speed up giant train's setup stage ... Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:34:32 +02:00
lars	1115451c8e	Make default checkpoint out_dir name reflect only non-default hyperparams ... Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs.	2026-07-29 11:21:02 +02:00
lars	539b6f61e1	Add test coverage for resolve_expert_dims ... Covers the default-config 0-sentinel inheritance path (the exact bug fixed by 969c5c6, previously untested since every router test in test_router.py passes expert_hidden_dim/expert_n_blocks explicitly), plus missing-key inheritance, full override, and partial override. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:44 +02:00
lars	ee29b9a303	router: seed EnergyRouter centers from data quantiles instead of a fixed linspace ... The 2026-07-22 rollout benchmark's router_gating diagnostic showed the 10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a roughly uniform z-normalized energy distribution, leaving experts heavily overlapping instead of partitioning the range. Add an optional centers_init kwarg (backward compatible, defaults to the old linspace) and have giant train estimate it from a reservoir sample of the real energy column, collected during the existing normalizer-fitting pass.	2026-07-27 13:11:16 +02:00
lars	61410ddee3	analyze: default run directory to <repo>/analysis_runs, gitignored ... giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.	2026-07-27 11:29:35 +02:00
lars	eb751d968d	transforms: pad legacy cond normalizers for pre-physical-conditioning checkpoints ... Checkpoints trained before commit 68fb99b (physical-property conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond normalizer, fit before build_cond_features grew the extra physical columns. Any inference against such a checkpoint under current code (predict/rollout/router_gating) crashed broadcasting a 15-wide cond_cont against an 8-wide mean/std. In "embedding" mode those physical columns are never read by ConditionEncoder, so padding the missing entries with mean=0/std=1 is a safe no-op. "physical" mode reads them directly, so a mismatch there still raises instead of silently normalizing garbage.	2026-07-27 11:06:48 +02:00
lars	dbd5c7e083	analyze: default condor docker image to alma9-gridjob ... mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9 not found). Switch the default to cverstege/alma9-gridjob, a modern EL9-based image.	2026-07-27 10:14:15 +02:00
lars	fa59443339	analyze: run condor compute jobs via .venv/bin/giant, not uv run ... uv isn't installed on the HTCondor worker docker image, so `uv run` fails there. giant is already an installed console script in the repo's uv-synced .venv, so exec it directly instead. write_submit now fails fast with a clear message if .venv/bin/giant is missing.	2026-07-27 10:03:02 +02:00
lars	e380400fe9	analyze: estimate per-job HTCondor walltime from chunk row count ... Each condor job's +RequestWalltime used to be one flat 3600s default for every (plot, chunk), regardless of how much data it actually streams over. `prep` now records each chunk's rollout+reference row count, and `giant/analysis/runtime_estimate.py` turns that into a per-job estimate: a per-spec (intercept, seconds/row) cost model fit by `scripts/profile_analysis_costs.py` against synthetic mock data on this machine, plus a fixed overhead placeholder (docker/uv/shared-fs startup — unmeasurable here, no /ceph access) and a single RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and the submit description references it via $(walltime) instead of a constant.	2026-07-27 09:47:44 +02:00
lars	86fc46b5a8	analyze: chunk per-plot aggregation across HTCondor jobs ... Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.	2026-07-27 09:24:19 +02:00
lars	60c2ca1985	analyze: add MoE router gating/share diagnostic plots ... New "model" family in the gallery: router_gating (mean soft gate weight vs. pre-step energy, showing the router's soft decision boundaries) and router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share by species / true physics process). Needs a live checkpoint's Router, so it's a documented exception to the rest of the package's polars/numpy-only contract; gracefully degrades to a placeholder for non-MoE checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:32:25 +02:00
lars	1115301eb9	test: replace prep(**_CTX) splat with a typed _prep helper ... ty correctly flagged this as unsound: _CTX's inferred dict[str, int] type doesn't rule out a "run_dir" key, which would silently bind to prep's own run_dir: str | Path | None parameter instead of falling through to **ctx_kwargs. Passing the context kwargs by name in a small test helper removes the ambiguity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 11:17:45 +02:00
lars	8ff70e3c87	analyze: drive prep/submit from the rollout YAML sidecar ... `giant analyze prep` / `submit` now take the `giant rollout` YAML sidecar as their only positional input instead of explicit --rollout/--reference/--out-dir. The YAML's `output`/`dataset` keys name the rollout parquet and its seed file (the reference truth), and the rest of the sidecar (checkpoint, geometry oracle, cutoffs) flows into every plot's gallery metadata. prep derives its own run directory next to the rollout parquet (<...>/analysis_<id>/) holding shared.json, run_meta.json, reduced/, plots/. compute-one and render now take just --run-dir / a run-dir argument and read the resolved paths + metadata from run_meta.json, so the condor wrapper no longer threads file paths. open_side scans a directory of reference shards via glob. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:56:36 +02:00
lars	f4c2545e8b	Rewrite analysis as streaming rollout-vs-reference plotting pipeline ... Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth diagnostics) with a lean giant/analysis/ package that compares one autoregressive `giant rollout` for a checkpoint against a held-out miniCaloSim reference file, and generates publication-styled plots in parallel on HTCondor. Rollout output and a raw reference file share a world-frame physical column subset under identical names, so the old ALR/local-frame decode machinery is gone — everything is world-frame mm/MeV. - sources.py: canonical LazyFrames, synthetic-termination-row filtering, the secondary view (rollout generation>0 tracks vs reference sec_*_list). - reduce.py: streaming primitives — a single hist1d group_by pass, per-event scalars, edep-weighted depth/transverse profiles, species share, leakage. - context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material group sets once into shared.json, so each compute job is one pass, no range scan (histogram efficiency). - catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg, material}, per-event totals, shower profiles, species/leakage, secondaries. - render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata. - condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit): one job per plot, compute/render split (workers polars-only, no LaTeX). Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover the reduce primitives, catalog id uniqueness + compute, condor submit, and a guarded render smoke test. Delete the two predict-diagnostics notebooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:38:09 +02:00
lars	44b0a92e67	Add WGAN-GP mode as a throwaway fast-eval experiment ... Adds --mode wgan alongside flow/ddpm: both stages get a WGAN-GP generator/critic pair (giant.model.wgan) instead of flow matching, so inference is a single forward pass per stage rather than a 10-step ODE integration — the fast-eval architecture noted in the roadmap. predict/rollout auto-detect the mode from the checkpoint's model_config. Best-checkpoint selection for wgan uses marginal-KL against the EMA generators every epoch, since a critic loss isn't a monotone quality signal. --router is not supported together with --mode wgan. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-22 11:20:28 +02:00
lars	06c9ad8e5f	Fix crashes in physical-property conditioning edge cases ... - make_seed_frontier only resolves particle mass/charge in "physical" mode, so "embedding"-mode rollouts no longer crash on a seed PDG code giant.particles can't resolve (the TERM_UNKNOWN_PDG gate now handles it). - nearest_known_pdg skips unresolvable candidate PDG codes instead of raising and killing the whole rollout/predict run. - predict/rollout fail with a clear message when a checkpoint predates the sec_phys normalizer, instead of a bare KeyError. - validate_marginals' phys_kl degrades to NaN (matching the energy_fraction_kl pattern) instead of crashing when a validated batch has zero secondaries on either side. - Correct CLAUDE.md's stale claim that the materials table is unfilled. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 10:36:25 +02:00
lars	68fb99bed8	Condition on material/particle physical properties instead of learned embeddings ... Adds model.conditioning = "physical" | "embedding": physical mode routes particle mass/charge and material Z_eff/A_eff/density/X0/lambda_int through small MLPs to replace the learned PDG/material embedding tables, so the surrogate generalizes to PDG codes/materials outside the training vocab instead of memorizing it. "embedding" stays available as the comparison baseline (old checkpoints without the key default to it). Stage 2 now regresses a secondary's mass/charge directly against a fixed physics-derived target instead of a learned/snapped embedding, and uses no snapping at inference — the model's raw predicted (mass, charge) is the secondary's physical identity, including for its own further rollout steps. A separate reporting-only nearest-known-PDG lookup (never fed back into the model) populates output pdg columns / the embedding-mode rollout fallback. giant/materials.py's table is populated with Geant4's own built-in NIST constants (Z_eff, A_eff, density, X0, lambda_int), extracted directly from the Geant4 11.4.1 build vendored in minicalosim via G4NistManager rather than hand-typed literature values. G4_LYSO is left unfilled: confirmed (both by runtime lookup and by searching minicalosim's history) that it's never actually a constructed Geant4 material there, only documentation/UI color-map text. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-17 15:12:54 +02:00
lars	81ec225b86	Reimplement rollout-vs-truth comparison on the streaming analysis module ... Merged in every non-analysis change from the MoE-prototype branch (routing, training, data pipeline, streaming rollout output), keeping this branch's lean streaming giant/analysis.py and rebuilding the rollout-vs-truth feature natively on it instead of resurrecting the old numpy SampleCollection path. - Add RolloutVsTruth, accepted anywhere Tier 1-3 functions take a predict-parquet source: decodes a giant rollout file and a held-out truth file into RAW_TARGET_NAMES space via a polars port of the forward local-frame rotation, fully streaming (no SampleCollection, no eager materialization). - Add compute_rollout_vs_truth_observables_pl for Tier 4, reusing EventObservables (now backed by independent real_table/gen_table to support unequal rollout/truth event counts) so every existing shower-observable plot function works unchanged for both one-step and full-rollout comparisons. - Update analysis/rollout_validation.ipynb to the new API and CLAUDE.md's architecture description; add test coverage for the new source type. - Fix a pre-existing return-type mismatch in giant.rollout.rollout() (found by `ty check`): the on_chunk summary-dict branch didn't match the declared dict[str, np.ndarray] return type, now expressed as a RolloutSummary TypedDict. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-17 12:32:30 +02:00
lars	6744d05948	Merge branch '4-prototype-a-mixture-of-experts-routing-tree-architecture' into analysis-streaming-rewrite ... # Conflicts: # giant/analysis.py # tests/test_analysis.py	2026-07-17 12:06:10 +02:00