189 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
lars	74343d3e48	Add data-integrity guards against silent NaN/Inf propagation and races ... - log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:29 +02:00
lars	a4c0443e01	Add bigger WGAN config (hidden_dim=512, n_blocks=6) ... Scaled-up variant of the already-trained wgan_h128_b4_physical.toml benchmark config, same training recipe (mode/epochs/lr/warmup), increased model capacity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 17:08:46 +02:00
lars	22fdca7697	Merge pull request 'Add opt-in straight-through Gumbel-softmax combine weights to MoE router' (#26) from feat/router-gumbel-softmax into master ... Reviewed-on: #26	2026-07-30 17:05:08 +02:00
lars	8065df896e	Scope wandb run config to only-active hyperparameters ... Router-only knobs (lambda_balance/lambda_proc/lambda_entropy/ gumbel_tau_start/_end) and WGAN-only knobs (n_critic/gp_weight) were being logged to wandb's top-level run config unconditionally, even for runs where routing or WGAN mode is off, implying hyperparameters from an inactive code path. Extract _wandb_run_config and only include each group when its gate is actually true (router.enabled / mode=="wgan"); the full model_config (with its router sub-dict) is still always logged in full, so no information is lost. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:46:19 +02:00
lars	5eec4c250a	Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming ... Extends default_out_dir_name's non-default-field convention to the router's new gumbel combine-weight flag and its learnable-knob toggles, so gumbel sweep configs (learn_centers on/off, learn_width, learn_temperature) resolve to distinguishable checkpoint directory names instead of colliding on the same r-<type><n> token. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:42:56 +02:00
lars	af2ee7c7ce	Add gumbel router configs sweeping learnable-knob combinations ... Extends the 10-expert EnergyRouter + physical-conditioning benchmark config with the new opt-in gumbel combine weights, isolating the learn_centers/learn_temperature axis: none, centers only, and centers+temperature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:40:02 +02:00
lars	b51eafcfa5	Add opt-in straight-through Gumbel-softmax combine weights to MoE router ... Trains the routed trunk's forward combination as a hard one-hot sample (matching eval-time top-1 dispatch exactly) while keeping a smooth gradient on the backward pass, targeting the train/eval mismatch identified as a likely contributor to experts overlapping instead of partitioning in the first energy-router rollout benchmark. Off by default (model.router.gumbel); existing routed configs/checkpoints are unaffected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 16:29:20 +02:00
lars	43cb6dd9ae	Merge pull request 'Add learnable per-expert width and shared temperature to EnergyRouter' (#25) from feat/router-learnable-width into master ... Reviewed-on: #25	2026-07-30 16:18:35 +02:00
lars	aa55c407ab	Merge branch 'master' into feat/router-learnable-width	2026-07-30 15:58:55 +02:00
lars	da5f54ea1c	Add learnable per-expert width and shared temperature to EnergyRouter ... EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 15:33:52 +02:00
lars	78297456e3	Merge pull request 'Store a quantile grid instead of a raw reservoir sample in the setup cache' (#24) from perf/quantile-energy-reservoir into master ... Reviewed-on: #24	2026-07-30 13:59:17 +02:00
lars	d656cf3109	Store a quantile grid instead of a raw reservoir sample in the setup cache ... NormalizerEntry.energy_reservoir_sample kept 100k raw energy values purely to seed EnergyRouter centers via np.quantile at load time, which alone accounted for most of the setup cache sidecar's ~2MB size (float32 values round-tripped through Python floats serialize at full double precision). Only a handful of quantile levels are ever read back, so collapse the sample to a fixed 1001-point quantile grid at save time and interpolate arbitrary levels from it at use time instead — about 100x smaller with negligible (<0.001) error on the levels that matter. Bumps the cache format version since old sidecars have no such grid to fall back on.	2026-07-30 13:32:46 +02:00
lars	de5db25e3f	Add giant new-run to scaffold a config.toml + run dir ahead of training ... Pulled forward from the not-yet-mergeable condor-gpu-train-rollout branch: `new-run` resolves CLI hyperparameter overrides into a full config.toml and run dir (reusing the existing default_out_dir_name collision-avoidance and a newly factored-out router-override helper shared with `train`), so a run can be prepared and reviewed before `giant train` actually kicks off. Also brings README up to date with the model/CLI as it actually stands (physical/embedding conditioning, WGAN/MoE-router modes, giant analyze, W&B, setup-stage caching), which had drifted back to describing the Phase-1 proof-of-concept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 12:56:22 +02:00
lars	1fd2625889	Merge pull request 'Offset event_id per file to avoid cross-file collisions' (#23) from fix/event-id-file-offset into master ... Reviewed-on: #23	2026-07-30 11:44:31 +02:00
lars	b944bba8fb	Offset event_id per file to avoid cross-file collisions ... Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 11:36:48 +02:00
lars	e288c3fe21	Merge pull request 'Cache giant train's setup stage in a sidecar file' (#21) from perf/setup-stage into master ... Reviewed-on: #21	2026-07-30 11:02:11 +02:00
lars	5aaf6cde4d	Merge branch 'master' into perf/setup-stage	2026-07-30 10:57:48 +02:00
lars	471a81b5e7	Add dwarf warm-cache to precompute the setup-stage sidecar ... Lets the vocab maps, event-id split index, and normalizer stats be warmed once for a dataset (right after `dwarf convert`, or before a `dwarf hparam-scan` sweep) without needing to also start training. Extracts the setup-stage logic out of giant/pipeline.py:run_train_job into a standalone run_setup_stage() (returning a SetupStageResult), reused by both run_train_job and the new dwarf command's scripts/warm_setup_cache.py — a behavior-preserving refactor, covered by the existing test_pipeline.py suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:53:31 +02:00
lars	26aa9d3fde	Pass --seed through to the train/val event split ... make_event_split() defaults to seed=42, and run_train_job was calling it without forwarding t["seed"] — so the configured --seed affected model init/EMA/etc. but not which events landed in train vs. val, which silently always used seed 42 regardless of --seed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:29:19 +02:00
lars	e7478c36fb	Cache giant train's setup stage in a sidecar file ... Building the pdg/material vocab maps, the process map, and fitting the Stage-1/Stage-2 normalizers all require scanning the training dataset before a single epoch runs, which is wasted work whenever the same data path is reused across runs (hyperparameter sweeps via `dwarf hparam-scan`, repeated manual training attempts, ...). Persist those setup-stage outputs to a JSON sidecar next to the input data (giant/data/setup_cache.py), validated by a file fingerprint plus fixed dimension constants and a manually-bumped format version before reuse, with a soft warning (not a hard invalidation) on a git-hash mismatch alone. Also derives n_train_steps instantly from cached per-event row counts instead of accumulating it during the normalizer scan, and always collects the energy-router reservoir sample while the cache is being populated (not only when the current run's router is energy-typed) so a later run enabling --router-type energy never needs to rescan just to seed expert centers. New --cache-setup/--no-cache-setup (default on) and --rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:28:37 +02:00
lars	9112625a08	Merge pull request 'Speed up giant train's setup stage' (#20) from perf/setup-stage into master ... Reviewed-on: #20	2026-07-29 14:02:27 +02:00
lars	84efbf5c2c	Merge branch 'master' into perf/setup-stage	2026-07-29 13:56:59 +02:00
lars	759b67a9e1	Speed up _WelfordAccumulator's per-chunk update ... The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:55:07 +02:00
lars	47a6c9db1f	Add regression coverage for vocab/process index-map builders ... build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:45:33 +02:00
lars	e331148afa	Merge pull request 'Make default checkpoint out_dir name reflect only non-default hyperparams' (#19) from feature/smart-outdir-naming into master ... Reviewed-on: #19	2026-07-29 13:38:16 +02:00
lars	09e4c765c7	Speed up giant train's setup stage ... Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:34:32 +02:00
lars	1115451c8e	Make default checkpoint out_dir name reflect only non-default hyperparams ... Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs.	2026-07-29 11:21:02 +02:00
lars	f2f89023d5	Merge pull request 'Feature/wandb integration' (#18) from feature/wandb-integration into master ... Reviewed-on: #18	2026-07-29 10:52:52 +02:00
lars	4c19072724	Skip empty-slice mean/std in sec phys validation print ... The per-dim print loop lacked the empty-array guard already used for the KL computation right above it and the sec-slot loop further down, so an all-zero-secondaries validation batch (e.g. early/unstable training) triggered numpy RuntimeWarnings from .mean()/.std() on empty arrays. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:49:12 +02:00
lars	539b6f61e1	Add test coverage for resolve_expert_dims ... Covers the default-config 0-sentinel inheritance path (the exact bug fixed by 969c5c6, previously untested since every router test in test_router.py passes expert_hidden_dim/expert_n_blocks explicitly), plus missing-key inheritance, full override, and partial override. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:44 +02:00
lars	f427d3384f	Timestamp default checkpoint dir to avoid W&B run-id collisions ... out_dir (and thus the W&B run id, which is derived from out_dir.name) was previously date-only, so two fresh runs on the same day with identical hyperparams silently shared one W&B run history. Default out_dir is now timestamped to the second. --resume without an explicit --out now reuses the checkpoint's own parent directory instead of recomputing a hyperparam-derived name, which both preserves the old continue-in-place behavior and fixes a latent bug where a resumed run with a changed hyperparam (e.g. --lr) would silently start writing to a new directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:40 +02:00
lars	bb699d41b2	Persist global_step across --resume so W&B step stays monotonic ... Previously global_step always reset to 0, even on --resume. Since the W&B run reattaches to the same run id on resume, logging with step=global_step after a restart passed step values below what was already recorded, silently dropping the resumed portion's metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:35 +02:00
lars	a986f96ba3	Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B ... Adds Router.gate_stats (per-router gate entropy + per-expert utilization), logged both per-batch (entropy only, train loop) and per-epoch (full stats, over the whole val set) — the router-collapse failure mode from the roadmap's rollout postmortem is now visible during training instead of only after a full rollout+analysis run. Also splits WGAN critic/ generator grad norms instead of summing them, logs critic LR, n_sec head accuracy, GPU peak memory + samples/sec, model parameter counts (in wandb.config), and an is_best flag — all wired into both metrics.csv and W&B. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:21:03 +02:00
lars	969c5c6e9a	Fix router experts silently ignoring --hidden-dim/--n-blocks ... expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed run always got fixed 128/3-wide experts no matter what --hidden-dim/ --n-blocks was passed. They now default to 0 ("unset"), which resolve_expert_dims() resolves by inheriting the model dims; an explicit override still works and now warns when it diverges from model.hidden_dim/n_blocks, since the checkpoint dir name won't reflect it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:34:25 +02:00
lars	29459ab1f7	Log batch-level metrics to W&B, not just per-epoch summaries ... giant train --wandb now also logs loss/grad_norm/lr every N optimizer steps (--wandb-log-every, default 50) so W&B shows within-epoch trends, not just one point per epoch. Both share global_step as a single monotonic step axis (wandb.Run.log requires step to never decrease across calls), which also fixes global_step previously only advancing in wgan mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:20:05 +02:00
lars	a05837f918	Apply ruff format ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:11:05 +02:00
lars	0778a61360	Add opt-in Weights & Biases logging for the training loop ... giant train --wandb logs the same per-epoch metrics already written to metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`) that nothing else depends on. A run's id is derived from the checkpoint out_dir so --resume reattaches to the existing run instead of starting a new one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:07:30 +02:00
lars	db0f12be58	Add configs for router energy (embedding/physical) and WGAN baseline runs ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 15:53:04 +02:00
lars	6208810089	Merge pull request 'Refactor the analysis plot creation with focus on rollout' (#16) from analysis-rollout-plots into master ... Reviewed-on: #16	2026-07-27 14:58:51 +02:00
lars	5594866eac	Merge remote-tracking branch 'origin/master' into analysis-rollout-plots ... # Conflicts: # analysis/rollout_validation.ipynb	2026-07-27 14:55:00 +02:00
lars	0f7febfd02	ci: set UV_LINK_MODE=copy to silence the cross-filesystem hardlink warning ... The bind-mounted uv cache and the job workspace are on different filesystems, so uv already falls back to copying installed files; this just tells it to do so directly instead of logging a hardlink-failed warning every job.	2026-07-27 14:40:01 +02:00
lars	19be455346	ci: re-pin UV_CACHE_DIR after setup-uv, which exports its own value regardless of enable-cache ... Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path (/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step right after setup-uv so it wins for the rest of the job.	2026-07-27 14:31:22 +02:00
lars	1787d26d1c	ci: stop setup-uv from overriding UV_CACHE_DIR ... setup-uv's default enable-cache: auto sets its own UV_CACHE_DIR (a tool-cache tmp path) as a later step, clobbering the workflow-level UV_CACHE_DIR that points at the bind-mounted persistent cache. Disable setup-uv's own cache handling so our mount stays in effect.	2026-07-27 14:22:49 +02:00
lars	0afa75ee30	ci: replace unsupported artifact sharing with a bind-mounted uv cache ... The self-hosted act_runner doesn't support upload/download-artifact, so drop that plumbing and instead run jobs in an explicit container with a persistent host directory mounted at /uv-cache (UV_CACHE_DIR), backed by valid_volumes on the runner. uv sync still runs per job but hits a warm local cache instead of re-downloading/building packages every time.	2026-07-27 14:18:54 +02:00
lars	08c76a9614	ci: share one uv sync across jobs, gate tests on lint+type-check, sync tag/version on release tags ... Uploads the synced .venv as an artifact from a single setup job instead of re-running uv sync (and re-downloading torch) in every job. Drops the build/publish job in favor of a lighter job that, on a pushed tag, checks the tag against the uv project version and — if they differ — bumps the version, commits it to master, and recreates the tag on the new commit.	2026-07-27 14:00:59 +02:00
lars	b8a4dc7d63	analyze: thread full model/training/rollout/dataset params to plots ... giant rollout now writes the checkpoint's complete model_config (incl. the router sub-dict), the sibling config.toml's [train]/[meta] sections, and every rollout CLI knob (weights, batch_size, escape_threshold, n_events, device, seed) into the YAML sidecar instead of a hand-picked subset. All of it flows through run_meta.json into each plot's own metadata.yaml for later comparison, while the figure subtitle itself shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router, epoch, best_val_loss, steps/noise_dim) via new_figure's params option. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 13:35:02 +02:00
lars	4feb9007f3	docs: note the EnergyRouter centers_init fix in the roadmap ... Records both identified contributors to the 2026-07-22 router divergence (missing lambda_balance and the uniform-linspace center init) and that the latter is now fixed, with next steps covering both for the retrain.	2026-07-27 13:14:49 +02:00
lars	ee29b9a303	router: seed EnergyRouter centers from data quantiles instead of a fixed linspace ... The 2026-07-22 rollout benchmark's router_gating diagnostic showed the 10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a roughly uniform z-normalized energy distribution, leaving experts heavily overlapping instead of partitioning the range. Add an optional centers_init kwarg (backward compatible, defaults to the old linspace) and have giant train estimate it from a reservoir sample of the real energy column, collected during the existing normalizer-fitting pass.	2026-07-27 13:11:16 +02:00
lars	641bbb0a68	docs: record first MoE router rollout benchmark result in the roadmap ... The 10-expert EnergyRouter checkpoint (lambda_balance=0.0) diverges badly from Geant4 on rollout (step granularity, secondary species, shower shape), and the router_gating diagnostic shows the experts heavily overlap rather than partitioning the pre-step energy domain. Recorded as "needs retraining with a different router config" rather than an abandoned direction. Full analysis in the knowledge base at experiments/giant-router-energy-rollout-validation.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 12:02:59 +02:00
lars	61410ddee3	analyze: default run directory to <repo>/analysis_runs, gitignored ... giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.	2026-07-27 11:29:35 +02:00