134 Commits

Author	SHA1	Message	Date
lars	8019a80563	Refactor train.py into giant/training/ around a metrics collector ... Every metric name used to exist in four places: the dict keys each StageTrainer returned, the hardcoded _metrics_fields() column list, the ~110-line metrics_row assembly in train(), and the tqdm/summary formatting. The two had to be kept in exact correspondence by hand or csv.DictWriter would raise. Each metric is now declared once, as a MetricSpec on the trainer that computes it. MetricsCollector derives the CSV header and W&B payload from those declarations and owns all accumulation, so train() no longer carries a running sum, and every isinstance(tr, WGANStageTrainer) branch is gone — replaced by four trainer hooks (batch_loss, summary, val_objective, supports_val_loss). giant/train.py (1875 lines) becomes giant/training/: trainers.py StageSpec + shared StageTrainer base + the two subclasses metrics.py MetricSpec, MetricsCollector stage2_inputs.py the pure AR/teacher-forcing tensor helpers, moved verbatim loop.py train() (225 lines, was ~514) + graceful shutdown checkpoint.py build/load, lifted out of train()'s closures The trainers shared ~15 identical constructor arguments and copy-pasted their cosine-warmup lambda, EMA setup, state_dict/load_state_dict, resume_lr and train_mode/eval_mode. StageSpec resolves one stage's config once (constructors go from 24 and 22 keyword arguments to (spec, model, device)), the base class holds the rest, and build_stage_trainers drops from ~100 lines to 15. Metric columns are renamed to a uniform stage/split/metric scheme (stage1/train/loss, stage2/train/d_loss, stage1/lr, stage1/router/entropy, val/loss, ...). Old metrics.csv files and W&B history are not comparable. The checkpoint format is unchanged. BEHAVIOR CHANGE — WGAN best-checkpoint selection. The old code meant to score a WGAN stage on its marginal KL, but the guard `{n: kl for n in wgan_names if n not in val_loss_per_stage}` could never fire: val_loss_per_stage was pre-seeded with 0.0 for every stage, so a WGAN stage contributed a flat 0.0 and the KL was written to metrics.csv without ever influencing best.pt. val_objective now returns it as intended. On the test harness's default flow+wgan config val_loss went from 2.182 (stage 1 only) to 15.137 (stage 1 + KL 12.954), and which epoch won changed. Runs before this commit picked their best checkpoint on the non-adversarial stages alone. Written up in docs/v0.3.0-followups.md. Verified: 699 tests pass; ruff, ruff format and ty clean. Baseline-vs- refactor metrics.csv compared across five configs (flow+wgan, AR+onehot, routed, both-flow, AR-flow) — every comparable value bit-identical except val/loss where the fix applies. Resume appends without a duplicate header and reproduces a HEAD worktree's per-epoch losses and LRs exactly across the resume boundary. A refactored last.pt loads through cli.py:_load_model_weights in both raw and ema modes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-07 17:03:20 +02:00
lars	da7cde3ef9	v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies ... Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 16:12:58 +02:00
lars	200c6d243b	v0.3.0 step 7: AttentionHistory (KV-cached) + scheduled/never teacher forcing ... AttentionHistory (giant/model/network.py) adds causal self-attention over the emitted-secondary prefix as the alternative to MarkovHistory, with a parallel forward() for training and an init_cache()/step() KV-cache path for sample.py's per-slot AR inference loop, wired into Stage2Autoregressive via history="attention". giant/train.py adds _stage2_tf_prob and _assemble_stage2_ar_inputs_scheduled, mixing ground-truth history with a detached sample_secondaries_ar self-sample per slot so teacher_forcing="scheduled"/"never" close the train/inference gap teacher_forcing="always" always avoided; wired into both stage-2 AR trainers. config.py's validate_config no longer rejects these two previously unimplemented schema values. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 13:15:56 +02:00
lars	93b19911f8	v0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode ... - giant/sample.py: fix every sampler's call convention against Stage1Model/Stage2OneShot's actual forward signatures (was still calling model(x, t, cond_cont, cond_cat) positionally); add sample_secondaries_ar (free-running AR loop, unsnapped history feature) and sample_stage1/sample_stage2/resolve_n_sec dispatch helpers that read each stage's generator_kind/decoder off the model instance itself. - giant/particles.py: decode_topn_class (argmax + other_policy) and decode_embedding_nearest (L1-snap + distance) turn a secondary's "onehot"/"embedding" type prediction into a concrete PDG. - giant/rollout.py: decode_secondary_identity routes all three particle_type.target values to real mass/charge; per-stage generator dispatch (drops the single shared `mode` string, adds ddpm support); L1DistCollector accumulates the §11.3 embedding-distance diagnostic. - giant/cli.py: drop the onehot/embedding-target rejection gate (narrowed to the still-unimplemented conditioning.particle/material.type=onehot axis); fix the dead model_cfg.get("mode") bug in predict/rollout. - giant/analysis/: new type_embedding_l1_distance PlotSpec, wired through the rollout YAML sidecar (no live-model call needed, unlike router_gating -- the histogram is already pre-aggregated at rollout time). - Un-xfail every test that was blocked on this step (test_rollout.py, test_flow.py, test_wgan.py, test_phase2.py, test_router.py, test_validate.py); add test_sample.py, test_type_embedding_distance.py. Known follow-up: giant/validate.py still unpacks the training val-batch as a stale 6-tuple and doesn't use the new per-stage dispatch, so marginal validation during training degrades gracefully with a warning rather than working -- not in this step's scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 10:37:57 +02:00
lars	c9d255b1c5	v0.3.0 step 5: Stage2Autoregressive (history=markov) + §11.4 grad instrumentation ... Replaces the Stage2Autoregressive stub with a real per-token secondary decoder: MarkovHistory summarizes the previous secondary, remaining-energy fraction and slot index round out the per-token conditioning, and the existing Trunk/MonolithicTrunk/RoutedTrunk machinery is reused unchanged by batching all K_MAX tokens together under teacher forcing (one parallel pass, no new trunk code). build_models/build_critics wire it in; the WGAN critic stays whole-sequence, so build_critics needs no AR-specific path. train.py's FlowDDPMStageTrainer/WGANStageTrainer gain a decoder branch, sharing optimizer/EMA/checkpoint machinery with the one-shot path. _assemble_stage2_real is now defined in terms of the new unflattened _assemble_stage2_ar_target helper, removing a near-duplicate branch. Also lands the §11.4 differentiability validation-obligation instrumentation (trunk-gradient norm from the particle-type slice vs. the continuous slices, for generator=wgan + particle_type.target=onehot) via backward hooks in _relax_onehot_type_slice, decoder-agnostic and surfaced as two new metrics.csv columns. This also fixes the standing regression where any config not explicitly overriding decoder="one_shot" crashed at build_models, since stage2_model.decoder defaults to "autoregressive" — confirmed by removing tests/test_pipeline.py's now-stale override so the default config runs end-to-end against real synthetic data. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 09:36:49 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	9112e845e0	v0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite ... Replaces train.py's single global training loop with a StageTrainer hierarchy (FlowDDPMStageTrainer, WGANStageTrainer) — one per active stage, each owning its own optimizer/LR schedule/EMA and reading only the shared batch tuple (stage 2 always teacher-forces on the ground-truth x1_s1, so stages never need each other's output at train time). Supports every stage1/stage2 generator combination, including the design doc's headline mixed case (stage1=flow + stage2=wgan) and its reverse, plus stage1-only/stage2-only ablation runs, routed+gumbel stages, and checkpoint save/resume. metrics.csv/wandb logging are stage-prefixed. validate_marginals calls are guarded with a one-time warning and a Wasserstein-magnitude fallback for wgan best-checkpoint selection, since giant/sample.py still assumes stage1 always owns n_sec_head (decision 1 moved it to stage 2 by default) — deferred to design doc step 6, not silently papered over. pipeline.py's run_setup_stage/run_train_job now read the new nested config directly; the dangling resolve_expert_dims call and the --mode wgan --router rejection are both gone (routed WGAN works). cli.py's train/new-run build correctly-shaped config overrides (architecture flags -> stage1_model only per the approved decision; --mode/--n-critic/--gp-weight/--critic-lr broadcast to both stages, matching migrate_config's own precedent and avoiding a regression on the common --mode case); predict/rollout's dangling build_models tuple-unpack is fixed; new-run now tags config_version, fixing a bug where a re-loaded v0.3 config.toml would have been silently corrupted by migrate_config mistaking it for v0.2. config.py's validate_config rejects mixed particle/material conditioning types for now (ConditionEncoder supports it, the data pipeline in giant/data/transforms.py doesn't yet). analysis/render.py and router_gating.py handle both the new nested model_config shape and legacy flat checkpoints. scripts/warm_setup_cache.py updated for run_setup_stage's new signature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 11:31:49 +02:00
lars	9ce55e5013	v0.3.0 step 2: network.py refactor to composable stage models ... Decomposes the ten permutation classes in giant/model/network.py into the reusable parts from docs/v0.3.0-design.md §5: ConditionEncoder (now independently configurable per particle/material axis), ContextAdapter, Trunk/MonolithicTrunk/RoutedTrunk/ExpertTrunk, and the stage classes Stage1Model/Stage2OneShot/CriticModel (Stage2Autoregressive stubbed, raises NotImplementedError until step 4/5). build_models/build_critics now return a dict keyed by stage and accept the new nested config shape, with routed WGAN reachable for the first time (the old --mode wgan --router rejection is gone) and stage2_model.router.tie_to_stage1 sharing a literal Router instance. A v0.2 checkpoint's flat model_config auto-migrates via _migrate_legacy_model_config + migrate_legacy_state_dict, preserving the n_sec_head's attachment to Stage1Model (legacy_owner="stage1", design doc §4.1). tests/test_migration_v02_v03.py proves this bit-identical against a frozen v0.2 snapshot (tests/legacy/network_v02_snapshot.py) for both flow and wgan, both conditioning modes. scripts/check_migration_v02_v03.py is the real-checkpoint counterpart for a portal machine with /ceph access. giant/model/schedule.py's flow-matching/DDPM loss helpers are updated to the new model-call convention (t as a keyword). giant/sample.py, giant/rollout.py, and giant/validate.py are not yet updated (deferred to design doc step 6) — their exercising tests are marked xfail with that reasoning rather than silently broken. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:55:29 +02:00
lars	eb6dd27406	v0.3.0 step 1: new nested config schema, v0.2 migration shim ... Replace the single global train.mode + [model] block with the four top-level blocks docs/v0.3.0-design.md specifies ([conditioning], [stage1_model], [stage2_model], [train]), so Stage 1 and Stage 2 can run independent generative objectives and Stage 2 can train standalone. - migrate_config translates old config.toml/checkpoint dicts on load, so nothing on /ceph goes dead; loudly rejects non-zero expert_hidden_dim/expert_n_blocks, which v0.3.0 no longer supports. - merge_cli_overrides/save_config generalize from one hardcoded nesting level (model.router) to arbitrary recursive depth. - default_out_dir_name candidates move to dotted paths against the new schema, with per-stage router/generator discriminators. - validate_config adds cross-block checks the per-block schema can't express (particle_type.target=embedding needs a matching conditioning mode, tie_to_stage1 needs an active stage 1, etc). - resolve_expert_dims is deleted (experts always inherit the stage's hidden_dim/n_res_blocks now) — pipeline.py/cli.py callers are left dangling on purpose, to be updated in the network.py/train.py steps that follow. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 10:15:20 +02:00
lars	c3e5956718	Resolve giant condor wrapper from the active venv, not a hardcoded path ... write_submit baked in cfg.repo_dir/.venv/bin/giant unconditionally, which breaks when submitting from a differently-named or non-default venv (e.g. --extra cuda). Prefer the giant executable next to sys.executable (the venv actually running the submit), falling back to repo_dir/.venv/bin/giant.	2026-08-03 14:46:54 +02:00
lars	dae6451203	Apply ruff format ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 14:04:17 +02:00
lars	ca3a2a3462	Fix CLI/tooling robustness gaps and dedupe the Conditioning enum ... - run_create_manifest gains --force; it previously overwrote an existing manifest (including holdout.manifest, which check_holdout_overlap exists specifically to protect) with no warning or backup on a second run. - _git_user_name only caught OSError, not subprocess.TimeoutExpired (a SubprocessError, not an OSError) — a slow/loaded shared portal machine could crash `dwarf bump-gen`/`bump-schema` instead of degrading to by=None as intended. - `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when the requested count exceeds ~1/4 of the machine's CPUs, matching the same shared-machine etiquette check added to giant train in the previous commit. - The Conditioning enum was independently redefined in both giant/cli.py and scripts/dwarf.py; moved to a single giant.config.Conditioning both now import, removing the drift risk of a third conditioning mode being added to one but not the other. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:49:09 +02:00
lars	ad1b8e7835	Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline ... - prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:43 +02:00
lars	ad0341a9d4	Fix training-loop checkpoint/resume and WGAN bugs ... - Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of in-progress weights before exiting mid-epoch — it previously broke out of the epoch loop before reaching the checkpoint-save block, contradicting its own printed "saving a checkpoint" message and losing all progress since the last completed epoch. Checkpoint-dict construction is factored into a shared _build_checkpoint() helper used by both the mid-epoch and end-of-epoch save paths. - WGAN LR-schedule steps_per_epoch used the wrong denominator (n_critic + 1 instead of n_critic), causing the schedule to exhaust early and LR to floor to 0 before training completed. - --critic-lr override was silently dropped on WGAN --resume (only the generator optimizer's LR was made authoritative again after load_state_dict; optimizer_d's was not). - WGAN secondary gradient-penalty forced x_hat/grad to zero for fully-masked rows (n_sec == 0, common in a shower), adding a constant ~1.0 bias into the batch-mean GP term; such rows are now excluded from the mean. - run_train_job warns (never blocks) when --num-workers exceeds ~1/4 of the machine's CPUs, per this repo's shared-portal-machine etiquette (see CLAUDE.md's Compute environment section). Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:22 +02:00
lars	5b63dfd588	Fix conditioning="physical" so it can actually generalize past training vocab ... The whole point of conditioning="physical" is generalizing to a species/material outside the training menu, but two independent code paths still hard-required training-vocab membership: - giant/data/transforms.py: build_cond_features unconditionally raised KeyError on an out-of-vocab pdg/material. _vectorized_map_lookup gains a strict=False mode (dummy index instead of raising), used only under conditioning="physical" where ConditionEncoder never reads cond_cat anyway; "embedding" mode is untouched and still raises, since cond_cat IS the conditioning signal there. - giant/rollout.py: the known_pdg termination gate still killed a track on step 1 for any pdg outside pdg_map, regardless of conditioning mode. Now skipped entirely under conditioning="physical". - giant/model/network.py: PdgRouter/ProcessRouter always build their own training-vocab nn.Embedding independent of conditioning, silently reintroducing the same limitation at the routing layer. build_models now raises loudly if conditioning="physical" is paired with either router type, rather than silently building a model that can't generalize the way it claims to. This unblocks the held-out-species/material generalization experiment against the multi-material dataset (see CLAUDE.md roadmap). Each fix has a regression test, including an end-to-end rollout test seeded with a resolvable-but-out-of-vocab PDG code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:59 +02:00
lars	74343d3e48	Add data-integrity guards against silent NaN/Inf propagation and races ... - log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:29 +02:00
lars	8065df896e	Scope wandb run config to only-active hyperparameters ... Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:46:19 +02:00
lars	5eec4c250a	Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming ... Extends default_out_dir_name's non-default-field convention to the router's new gumbel combine-weight flag and its learnable-knob toggles, so gumbel sweep configs (learn_centers on/off, learn_width, learn_temperature) resolve to distinguishable checkpoint directory names instead of colliding on the same r-<type><n> token. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:42:56 +02:00
lars	b51eafcfa5	Add opt-in straight-through Gumbel-softmax combine weights to MoE router ... Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:29:20 +02:00
lars	da5f54ea1c	Add learnable per-expert width and shared temperature to EnergyRouter ... EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 15:33:52 +02:00
lars	d656cf3109	Store a quantile grid instead of a raw reservoir sample in the setup cache ... NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely to seed EnergyRouter centers via np.quantile at load time, which alone accounted for most of the setup cache sidecar's ~2MB size (float32 values round-tripped through Python floats serialize at full double precision). Only a handful of quantile levels are ever read back, so collapse the sample to a fixed 1001-point quantile grid at save time and interpolate arbitrary levels from it at use time instead — about 100x smaller with negligible (<0.001) error on the levels that matter. Bumps the cache format version since old sidecars have no such grid to fall back on.	2026-07-30 13:32:46 +02:00
lars	de5db25e3f	Add giant new-run to scaffold a config.toml + run dir ahead of training ... Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch: `new-run` resolves CLI hyperparameter overrides into a full config.toml and run dir (reusing the existing default_out_dir_name collision-avoidance and a newly factored-out router-override helper shared with `train`), so a run can be prepared and reviewed before `giant train` actually kicks off. Also brings README up to date with the model/CLI as it actually stands (physical/embedding conditioning, WGAN/MoE-router modes, giant analyze, W&B, setup-stage caching), which had drifted back to describing the Phase-1 proof-of-concept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 12:56:22 +02:00
lars	b944bba8fb	Offset event_id per file to avoid cross-file collisions ... Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 11:36:48 +02:00
lars	471a81b5e7	Add dwarf warm-cache to precompute the setup-stage sidecar ... Lets the vocab maps, event-id split index, and normalizer stats be warmed once for a dataset (right after `dwarf convert`, or before a `dwarf hparam-scan` sweep) without needing to also start training. Extracts the setup-stage logic out of giant/pipeline.py:run_train_job into a standalone run_setup_stage() (returning a SetupStageResult), reused by both run_train_job and the new dwarf command's scripts/warm_setup_cache.py — a behavior-preserving refactor, covered by the existing test_pipeline.py suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:53:31 +02:00
lars	26aa9d3fde	Pass --seed through to the train/val event split ... make_event_split() defaults to seed=42, and run_train_job was calling it without forwarding t["seed"] — so the configured --seed affected model init/EMA/etc. but not which events landed in train vs. val, which silently always used seed 42 regardless of --seed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:29:19 +02:00
lars	e7478c36fb	Cache giant train's setup stage in a sidecar file ... Building the pdg/material vocab maps, the process map, and fitting the Stage-1/Stage-2 normalizers all require scanning the training dataset before a single epoch runs, which is wasted work whenever the same data path is reused across runs (hyperparameter sweeps via `dwarf hparam-scan`, repeated manual training attempts, ...). Persist those setup-stage outputs to a JSON sidecar next to the input data (giant/data/setup_cache.py), validated by a file fingerprint plus fixed dimension constants and a manually-bumped format version before reuse, with a soft warning (not a hard invalidation) on a git-hash mismatch alone. Also derives n_train_steps instantly from cached per-event row counts instead of accumulating it during the normalizer scan, and always collects the energy-router reservoir sample while the cache is being populated (not only when the current run's router is energy-typed) so a later run enabling --router-type energy never needs to rescan just to seed expert centers. New --cache-setup/--no-cache-setup (default on) and --rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:28:37 +02:00
lars	84efbf5c2c	Merge branch 'master' into perf/setup-stage	2026-07-29 13:56:59 +02:00
lars	759b67a9e1	Speed up _WelfordAccumulator's per-chunk update ... The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:55:07 +02:00
lars	09e4c765c7	Speed up giant train's setup stage ... Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:34:32 +02:00
lars	1115451c8e	Make default checkpoint out_dir name reflect only non-default hyperparams ... Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs.	2026-07-29 11:21:02 +02:00
lars	4c19072724	Skip empty-slice mean/std in sec phys validation print ... The per-dim print loop lacked the empty-array guard already used for the KL computation right above it and the sec-slot loop further down, so an all-zero-secondaries validation batch (e.g. early/unstable training) triggered numpy RuntimeWarnings from .mean()/.std() on empty arrays. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:49:12 +02:00
lars	f427d3384f	Timestamp default checkpoint dir to avoid W&B run-id collisions ... out_dir (and thus the W&B run id, which is derived from out_dir.name) was previously date-only, so two fresh runs on the same day with identical hyperparams silently shared one W&B run history. Default out_dir is now timestamped to the second. --resume without an explicit --out now reuses the checkpoint's own parent directory instead of recomputing a hyperparam-derived name, which both preserves the old continue-in-place behavior and fixes a latent bug where a resumed run with a changed hyperparam (e.g. --lr) would silently start writing to a new directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:40 +02:00
lars	bb699d41b2	Persist global_step across --resume so W&B step stays monotonic ... Previously global_step always reset to 0, even on --resume. Since the W&B run reattaches to the same run id on resume, logging with step=global_step after a restart passed step values below what was already recorded, silently dropping the resumed portion's metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:35 +02:00
lars	a986f96ba3	Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B ... Adds Router.gate_stats (per-router gate entropy + per-expert utilization), logged both per-batch (entropy only, train loop) and per-epoch (full stats, over the whole val set) — the router-collapse failure mode from the roadmap's rollout postmortem is now visible during training instead of only after a full rollout+analysis run. Also splits WGAN critic/ generator grad norms instead of summing them, logs critic LR, n_sec head accuracy, GPU peak memory + samples/sec, model parameter counts (in wandb.config), and an is_best flag — all wired into both metrics.csv and W&B. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:21:03 +02:00
lars	969c5c6e9a	Fix router experts silently ignoring --hidden-dim/--n-blocks ... expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed run always got fixed 128/3-wide experts no matter what --hidden-dim/ --n-blocks was passed. They now default to 0 ("unset"), which resolve_expert_dims() resolves by inheriting the model dims; an explicit override still works and now warns when it diverges from model.hidden_dim/n_blocks, since the checkpoint dir name won't reflect it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:34:25 +02:00
lars	29459ab1f7	Log batch-level metrics to W&B, not just per-epoch summaries ... giant train --wandb now also logs loss/grad_norm/lr every N optimizer steps (--wandb-log-every, default 50) so W&B shows within-epoch trends, not just one point per epoch. Both share global_step as a single monotonic step axis (wandb.Run.log requires step to never decrease across calls), which also fixes global_step previously only advancing in wgan mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:20:05 +02:00
lars	a05837f918	Apply ruff format ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:11:05 +02:00
lars	0778a61360	Add opt-in Weights & Biases logging for the training loop ... giant train --wandb logs the same per-epoch metrics already written to metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`) that nothing else depends on. A run's id is derived from the checkpoint out_dir so --resume reattaches to the existing run instead of starting a new one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:07:30 +02:00
lars	b8a4dc7d63	analyze: thread full model/training/rollout/dataset params to plots ... giant rollout now writes the checkpoint's complete model_config (incl. the router sub-dict), the sibling config.toml's [train]/[meta] sections, and every rollout CLI knob (weights, batch_size, escape_threshold, n_events, device, seed) into the YAML sidecar instead of a hand-picked subset. All of it flows through run_meta.json into each plot's own metadata.yaml for later comparison, while the figure subtitle itself shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router, epoch, best_val_loss, steps/noise_dim) via new_figure's params option. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 13:35:02 +02:00
lars	ee29b9a303	router: seed EnergyRouter centers from data quantiles instead of a fixed linspace ... The 2026-07-22 rollout benchmark's router_gating diagnostic showed the 10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a roughly uniform z-normalized energy distribution, leaving experts heavily overlapping instead of partitioning the range. Add an optional centers_init kwarg (backward compatible, defaults to the old linspace) and have giant train estimate it from a reservoir sample of the real energy column, collected during the existing normalizer-fitting pass.	2026-07-27 13:11:16 +02:00
lars	61410ddee3	analyze: default run directory to <repo>/analysis_runs, gitignored ... giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.	2026-07-27 11:29:35 +02:00
lars	a88b21ef70	style: ruff format	2026-07-27 11:19:06 +02:00
lars	eb751d968d	transforms: pad legacy cond normalizers for pre-physical-conditioning checkpoints ... Checkpoints trained before commit 68fb99b (physical-property conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond normalizer, fit before build_cond_features grew the extra physical columns. Any inference against such a checkpoint under current code (predict/rollout/router_gating) crashed broadcasting a 15-wide cond_cont against an 8-wide mean/std. In "embedding" mode those physical columns are never read by ConditionEncoder, so padding the missing entries with mean=0/std=1 is a safe no-op. "physical" mode reads them directly, so a mismatch there still raises instead of silently normalizing garbage.	2026-07-27 11:06:48 +02:00
lars	4d6101dcd7	analyze: recalibrate condor walltime model from real cluster timings ... The prior _COST_MODEL/_FIXED_OVERHEAD_S were fit only against local synthetic benchmarks (up to 2M rows/side), which can't see docker pull or real /ceph read latency and wildly overestimated real jobs (~1200-1800s predicted vs 50-320s median actual, from condor_history on production run 563f5ee3, --chunks 4, ~254M total rows). Refit each spec's per-row rate through the origin against its median real wall-clock time (not max, to avoid baking a few /ceph-contention spikes into a rate that would then wrongly scale with dataset size), and raised RUNTIME_SAFETY_MARGIN to compensate for that same contention risk instead.	2026-07-27 11:06:40 +02:00
lars	c7194701f6	analyze: raise default condor job memory request to 8192 MB ... 4096 MB was too tight: a resubmitted run held ~31 jobs spread evenly across nearly every plot family and chunk index with "Docker job has gone over memory limit of 4224 Mb", not one specific spec, so the generic per-chunk data footprint needed more headroom.	2026-07-27 10:31:46 +02:00
lars	dbd5c7e083	analyze: default condor docker image to alma9-gridjob ... mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9 not found). Switch the default to cverstege/alma9-gridjob, a modern EL9-based image.	2026-07-27 10:14:15 +02:00
lars	fa59443339	analyze: run condor compute jobs via .venv/bin/giant, not uv run ... uv isn't installed on the HTCondor worker docker image, so `uv run` fails there. giant is already an installed console script in the repo's uv-synced .venv, so exec it directly instead. write_submit now fails fast with a clear message if .venv/bin/giant is missing.	2026-07-27 10:03:02 +02:00
lars	e380400fe9	analyze: estimate per-job HTCondor walltime from chunk row count ... Each condor job's +RequestWalltime used to be one flat 3600s default for every (plot, chunk), regardless of how much data it actually streams over. `prep` now records each chunk's rollout+reference row count, and `giant/analysis/runtime_estimate.py` turns that into a per-job estimate: a per-spec (intercept, seconds/row) cost model fit by `scripts/profile_analysis_costs.py` against synthetic mock data on this machine, plus a fixed overhead placeholder (docker/uv/shared-fs startup — unmeasurable here, no /ceph access) and a single RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and the submit description references it via $(walltime) instead of a constant.	2026-07-27 09:47:44 +02:00
lars	85d3914a4d	analyze: expose bin/pdg options on analyze submit ... `submit` calls `prep` internally but only forwarded --chunks, so a condor run could never use non-default energy-bins/bins/top-pdg.	2026-07-27 09:30:54 +02:00
lars	86fc46b5a8	analyze: chunk per-plot aggregation across HTCondor jobs ... Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.	2026-07-27 09:24:19 +02:00