200 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
lars	376bdb9d08	Refine v0.3.0 design: defaults, deferred scope, open questions ... Config defaults: dropout 0.1 -> 0.0, wandb false -> true. The v0.3.0 work is a sequence of architecture comparisons, and an unlogged run is not comparable, so W&B is on unless explicitly disabled. Restructure the open-questions section into settled / deferred / tracked / still-open, since most of it is now decided: - other_policy three-way switch and separate flow/ddpm sub-tables are approved as specified. - stop_token is schema-valid but raises "not implemented in v0.3.0"; charge conservation gets no key at all and is left deliberately undesigned, to be worked out on its own terms rather than pre-shaped by this refactor. The speculative charge-mask sketch is removed. - Logging the L1 decode-distance distribution under target = "embedding" becomes tracked implementation work, landing with the rollout decode. - estimate_batch_size recalibration becomes implementation step 8, last: the activation-memory profile is not knowable until the AR trunk and history encoder are final, so it is measured on real hardware with the example configs rather than guessed. Only the differentiability question for Jan remains genuinely open. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-06 09:54:48 +02:00
lars	f390884f67	Add v0.3.0 design doc: Stage-2 autoregressive redesign ... Design contract for the v0.3.0 config break and network.py refactor, following the 2026-08-04 meeting with Jan. Not implemented yet. The 2026-08-03 WGAN rollout benchmark failed specifically at the secondary-species level (zero photons, ~4M hallucinated -14 muon antineutrinos). The response pivots Stage 2 to autoregressive generation in descending-energy order with teacher forcing, and reverts the particle type to a categorical representation. That needs a config break: [conditioning] / [stage1_model] / [stage2_model] / [train] replace the single global train.mode and [model] block, so per-stage generators (stage1 flow + stage2 wgan), stage-2-only training, and one-shot-vs-autoregressive comparison all become expressible. The particle and material conditioning axes are configured independently and mix freely, each with physical / embedding / onehot modes; the stage-2 type target mirrors the same three names, with conditioning.particle.emb_dim sizing both so the two share one class map. network.py collapses from ten permutation classes (stage x objective x routed) into composable parts — encoder x trunk x objective — which also makes routed WGAN work for the first time; it was only ever rejected because no routed WGAN generator class existed. The doc specifies every config option, the v0.2 migration (shim for both configs and checkpoints, gated on a bit-identical output diff), the refactor, and the implementation order. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-04 15:25:32 +02:00
lars	2fe887b49a	Bump version to 0.2.0 ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> v0.2.0	2026-08-04 14:01:33 +02:00
lars	803aae364e	format: Format tests/test_condor.py according to ruff styling	2026-08-03 14:57:45 +02:00
lars	057d637080	Fix test_write_submit_requires_synced_venv for active-venv resolution ... The venv-detection change in condor.py now checks for a giant binary next to sys.executable before falling back to repo_dir/.venv/bin/giant, and the test's own pytest venv has one — so the expected FileNotFoundError never fired. Monkeypatch sys.executable to a nonexistent path so the test exercises the fallback with neither location populated.	2026-08-03 14:52:41 +02:00
lars	c3e5956718	Resolve giant condor wrapper from the active venv, not a hardcoded path ... write_submit baked in cfg.repo_dir/.venv/bin/giant unconditionally, which breaks when submitting from a differently-named or non-default venv (e.g. --extra cuda). Prefer the giant executable next to sys.executable (the venv actually running the submit), falling back to repo_dir/.venv/bin/giant.	2026-08-03 14:46:54 +02:00
lars	dae6451203	Apply ruff format ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 14:04:17 +02:00
lars	ca3a2a3462	Fix CLI/tooling robustness gaps and dedupe the Conditioning enum ... - run_create_manifest gains --force; it previously overwrote an existing manifest (including holdout.manifest, which check_holdout_overlap exists specifically to protect) with no warning or backup on a second run. - _git_user_name only caught OSError, not subprocess.TimeoutExpired (a SubprocessError, not an OSError) — a slow/loaded shared portal machine could crash `dwarf bump-gen`/`bump-schema` instead of degrading to by=None as intended. - `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when the requested count exceeds ~1/4 of the machine's CPUs, matching the same shared-machine etiquette check added to giant train in the previous commit. - The Conditioning enum was independently redefined in both giant/cli.py and scripts/dwarf.py; moved to a single giant.config.Conditioning both now import, removing the drift risk of a third conditioning mode being added to one but not the other. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:49:09 +02:00
lars	ad1b8e7835	Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline ... - prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:43 +02:00
lars	ad0341a9d4	Fix training-loop checkpoint/resume and WGAN bugs ... - Graceful shutdown (SIGINT/SIGTERM) now actually saves a checkpoint of in-progress weights before exiting mid-epoch — it previously broke out of the epoch loop before reaching the checkpoint-save block, contradicting its own printed "saving a checkpoint" message and losing all progress since the last completed epoch. Checkpoint-dict construction is factored into a shared _build_checkpoint() helper used by both the mid-epoch and end-of-epoch save paths. - WGAN LR-schedule steps_per_epoch used the wrong denominator (n_critic + 1 instead of n_critic), causing the schedule to exhaust early and LR to floor to 0 before training completed. - --critic-lr override was silently dropped on WGAN --resume (only the generator optimizer's LR was made authoritative again after load_state_dict; optimizer_d's was not). - WGAN secondary gradient-penalty forced x_hat/grad to zero for fully-masked rows (n_sec == 0, common in a shower), adding a constant ~1.0 bias into the batch-mean GP term; such rows are now excluded from the mean. - run_train_job warns (never blocks) when --num-workers exceeds ~1/4 of the machine's CPUs, per this repo's shared-portal-machine etiquette (see CLAUDE.md's Compute environment section). Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:22 +02:00
lars	5b63dfd588	Fix conditioning="physical" so it can actually generalize past training vocab ... The whole point of conditioning="physical" is generalizing to a species/material outside the training menu, but two independent code paths still hard-required training-vocab membership: - giant/data/transforms.py: build_cond_features unconditionally raised KeyError on an out-of-vocab pdg/material. _vectorized_map_lookup gains a strict=False mode (dummy index instead of raising), used only under conditioning="physical" where ConditionEncoder never reads cond_cat anyway; "embedding" mode is untouched and still raises, since cond_cat IS the conditioning signal there. - giant/rollout.py: the known_pdg termination gate still killed a track on step 1 for any pdg outside pdg_map, regardless of conditioning mode. Now skipped entirely under conditioning="physical". - giant/model/network.py: PdgRouter/ProcessRouter always build their own training-vocab nn.Embedding independent of conditioning, silently reintroducing the same limitation at the routing layer. build_models now raises loudly if conditioning="physical" is paired with either router type, rather than silently building a model that can't generalize the way it claims to. This unblocks the held-out-species/material generalization experiment against the multi-material dataset (see CLAUDE.md roadmap). Each fix has a regression test, including an end-to-end rollout test seeded with a resolvable-but-out-of-vocab PDG code. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:59 +02:00
lars	74343d3e48	Add data-integrity guards against silent NaN/Inf propagation and races ... - log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:29 +02:00
lars	a4c0443e01	Add bigger WGAN config (hidden_dim=512, n_blocks=6) ... Scaled-up variant of the already-trained wgan_h128_b4_physical.toml benchmark config, same training recipe (mode/epochs/lr/warmup), increased model capacity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 17:08:46 +02:00
lars	22fdca7697	Merge pull request 'Add opt-in straight-through Gumbel-softmax combine weights to MoE router' (#26) from feat/router-gumbel-softmax into master ... Reviewed-on: #26	2026-07-30 17:05:08 +02:00
lars	8065df896e	Scope wandb run config to only-active hyperparameters ... Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:46:19 +02:00
lars	5eec4c250a	Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming ... Extends default_out_dir_name's non-default-field convention to the router's new gumbel combine-weight flag and its learnable-knob toggles, so gumbel sweep configs (learn_centers on/off, learn_width, learn_temperature) resolve to distinguishable checkpoint directory names instead of colliding on the same r-<type><n> token. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:42:56 +02:00
lars	af2ee7c7ce	Add gumbel router configs sweeping learnable-knob combinations ... Extends the 10-expert EnergyRouter + physical-conditioning benchmark config with the new opt-in gumbel combine weights, isolating the learn_centers/learn_temperature axis: none, centers only, and centers+temperature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:40:02 +02:00
lars	b51eafcfa5	Add opt-in straight-through Gumbel-softmax combine weights to MoE router ... Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:29:20 +02:00
lars	43cb6dd9ae	Merge pull request 'Add learnable per-expert width and shared temperature to EnergyRouter' (#25) from feat/router-learnable-width into master ... Reviewed-on: #25	2026-07-30 16:18:35 +02:00
lars	aa55c407ab	Merge branch 'master' into feat/router-learnable-width	2026-07-30 15:58:55 +02:00
lars	da5f54ea1c	Add learnable per-expert width and shared temperature to EnergyRouter ... EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 15:33:52 +02:00
lars	78297456e3	Merge pull request 'Store a quantile grid instead of a raw reservoir sample in the setup cache' (#24) from perf/quantile-energy-reservoir into master ... Reviewed-on: #24	2026-07-30 13:59:17 +02:00
lars	d656cf3109	Store a quantile grid instead of a raw reservoir sample in the setup cache ... NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely to seed EnergyRouter centers via np.quantile at load time, which alone accounted for most of the setup cache sidecar's ~2MB size (float32 values round-tripped through Python floats serialize at full double precision). Only a handful of quantile levels are ever read back, so collapse the sample to a fixed 1001-point quantile grid at save time and interpolate arbitrary levels from it at use time instead — about 100x smaller with negligible (<0.001) error on the levels that matter. Bumps the cache format version since old sidecars have no such grid to fall back on.	2026-07-30 13:32:46 +02:00
lars	de5db25e3f	Add giant new-run to scaffold a config.toml + run dir ahead of training ... Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch: `new-run` resolves CLI hyperparameter overrides into a full config.toml and run dir (reusing the existing default_out_dir_name collision-avoidance and a newly factored-out router-override helper shared with `train`), so a run can be prepared and reviewed before `giant train` actually kicks off. Also brings README up to date with the model/CLI as it actually stands (physical/embedding conditioning, WGAN/MoE-router modes, giant analyze, W&B, setup-stage caching), which had drifted back to describing the Phase-1 proof-of-concept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 12:56:22 +02:00
lars	1fd2625889	Merge pull request 'Offset event_id per file to avoid cross-file collisions' (#23) from fix/event-id-file-offset into master ... Reviewed-on: #23	2026-07-30 11:44:31 +02:00
lars	b944bba8fb	Offset event_id per file to avoid cross-file collisions ... Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 11:36:48 +02:00
lars	e288c3fe21	Merge pull request 'Cache giant train's setup stage in a sidecar file' (#21) from perf/setup-stage into master ... Reviewed-on: #21	2026-07-30 11:02:11 +02:00
lars	5aaf6cde4d	Merge branch 'master' into perf/setup-stage	2026-07-30 10:57:48 +02:00
lars	471a81b5e7	Add dwarf warm-cache to precompute the setup-stage sidecar ... Lets the vocab maps, event-id split index, and normalizer stats be warmed once for a dataset (right after `dwarf convert`, or before a `dwarf hparam-scan` sweep) without needing to also start training. Extracts the setup-stage logic out of giant/pipeline.py:run_train_job into a standalone run_setup_stage() (returning a SetupStageResult), reused by both run_train_job and the new dwarf command's scripts/warm_setup_cache.py — a behavior-preserving refactor, covered by the existing test_pipeline.py suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:53:31 +02:00
lars	26aa9d3fde	Pass --seed through to the train/val event split ... make_event_split() defaults to seed=42, and run_train_job was calling it without forwarding t["seed"] — so the configured --seed affected model init/EMA/etc. but not which events landed in train vs. val, which silently always used seed 42 regardless of --seed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:29:19 +02:00
lars	e7478c36fb	Cache giant train's setup stage in a sidecar file ... Building the pdg/material vocab maps, the process map, and fitting the Stage-1/Stage-2 normalizers all require scanning the training dataset before a single epoch runs, which is wasted work whenever the same data path is reused across runs (hyperparameter sweeps via `dwarf hparam-scan`, repeated manual training attempts, ...). Persist those setup-stage outputs to a JSON sidecar next to the input data (giant/data/setup_cache.py), validated by a file fingerprint plus fixed dimension constants and a manually-bumped format version before reuse, with a soft warning (not a hard invalidation) on a git-hash mismatch alone. Also derives n_train_steps instantly from cached per-event row counts instead of accumulating it during the normalizer scan, and always collects the energy-router reservoir sample while the cache is being populated (not only when the current run's router is energy-typed) so a later run enabling --router-type energy never needs to rescan just to seed expert centers. New --cache-setup/--no-cache-setup (default on) and --rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:28:37 +02:00
lars	9112625a08	Merge pull request 'Speed up giant train's setup stage' (#20) from perf/setup-stage into master ... Reviewed-on: #20	2026-07-29 14:02:27 +02:00
lars	84efbf5c2c	Merge branch 'master' into perf/setup-stage	2026-07-29 13:56:59 +02:00
lars	759b67a9e1	Speed up _WelfordAccumulator's per-chunk update ... The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:55:07 +02:00
lars	47a6c9db1f	Add regression coverage for vocab/process index-map builders ... build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:45:33 +02:00
lars	e331148afa	Merge pull request 'Make default checkpoint out_dir name reflect only non-default hyperparams' (#19) from feature/smart-outdir-naming into master ... Reviewed-on: #19	2026-07-29 13:38:16 +02:00
lars	09e4c765c7	Speed up giant train's setup stage ... Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:34:32 +02:00
lars	1115451c8e	Make default checkpoint out_dir name reflect only non-default hyperparams ... Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs.	2026-07-29 11:21:02 +02:00
lars	f2f89023d5	Merge pull request 'Feature/wandb integration' (#18) from feature/wandb-integration into master ... Reviewed-on: #18	2026-07-29 10:52:52 +02:00
lars	4c19072724	Skip empty-slice mean/std in sec phys validation print ... The per-dim print loop lacked the empty-array guard already used for the KL computation right above it and the sec-slot loop further down, so an all-zero-secondaries validation batch (e.g. early/unstable training) triggered numpy RuntimeWarnings from .mean()/.std() on empty arrays. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:49:12 +02:00
lars	539b6f61e1	Add test coverage for resolve_expert_dims ... Covers the default-config 0-sentinel inheritance path (the exact bug fixed by 969c5c6, previously untested since every router test in test_router.py passes expert_hidden_dim/expert_n_blocks explicitly), plus missing-key inheritance, full override, and partial override. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:44 +02:00
lars	f427d3384f	Timestamp default checkpoint dir to avoid W&B run-id collisions ... out_dir (and thus the W&B run id, which is derived from out_dir.name) was previously date-only, so two fresh runs on the same day with identical hyperparams silently shared one W&B run history. Default out_dir is now timestamped to the second. --resume without an explicit --out now reuses the checkpoint's own parent directory instead of recomputing a hyperparam-derived name, which both preserves the old continue-in-place behavior and fixes a latent bug where a resumed run with a changed hyperparam (e.g. --lr) would silently start writing to a new directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:40 +02:00
lars	bb699d41b2	Persist global_step across --resume so W&B step stays monotonic ... Previously global_step always reset to 0, even on --resume. Since the W&B run reattaches to the same run id on resume, logging with step=global_step after a restart passed step values below what was already recorded, silently dropping the resumed portion's metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:35 +02:00
lars	a986f96ba3	Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B ... Adds Router.gate_stats (per-router gate entropy + per-expert utilization), logged both per-batch (entropy only, train loop) and per-epoch (full stats, over the whole val set) — the router-collapse failure mode from the roadmap's rollout postmortem is now visible during training instead of only after a full rollout+analysis run. Also splits WGAN critic/ generator grad norms instead of summing them, logs critic LR, n_sec head accuracy, GPU peak memory + samples/sec, model parameter counts (in wandb.config), and an is_best flag — all wired into both metrics.csv and W&B. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:21:03 +02:00
lars	969c5c6e9a	Fix router experts silently ignoring --hidden-dim/--n-blocks ... expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed run always got fixed 128/3-wide experts no matter what --hidden-dim/ --n-blocks was passed. They now default to 0 ("unset"), which resolve_expert_dims() resolves by inheriting the model dims; an explicit override still works and now warns when it diverges from model.hidden_dim/n_blocks, since the checkpoint dir name won't reflect it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:34:25 +02:00
lars	29459ab1f7	Log batch-level metrics to W&B, not just per-epoch summaries ... giant train --wandb now also logs loss/grad_norm/lr every N optimizer steps (--wandb-log-every, default 50) so W&B shows within-epoch trends, not just one point per epoch. Both share global_step as a single monotonic step axis (wandb.Run.log requires step to never decrease across calls), which also fixes global_step previously only advancing in wgan mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:20:05 +02:00
lars	a05837f918	Apply ruff format ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:11:05 +02:00
lars	0778a61360	Add opt-in Weights & Biases logging for the training loop ... giant train --wandb logs the same per-epoch metrics already written to metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`) that nothing else depends on. A run's id is derived from the checkpoint out_dir so --resume reattaches to the existing run instead of starting a new one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:07:30 +02:00
lars	db0f12be58	Add configs for router energy (embedding/physical) and WGAN baseline runs ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 15:53:04 +02:00
lars	6208810089	Merge pull request 'Refactor the analysis plot creation with focus on rollout' (#16) from analysis-rollout-plots into master ... Reviewed-on: #16	2026-07-27 14:58:51 +02:00