174 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
lars	e288c3fe21	Merge pull request 'Cache giant train's setup stage in a sidecar file' (#21) from perf/setup-stage into master ... Reviewed-on: #21	2026-07-30 11:02:11 +02:00
lars	5aaf6cde4d	Merge branch 'master' into perf/setup-stage	2026-07-30 10:57:48 +02:00
lars	471a81b5e7	Add dwarf warm-cache to precompute the setup-stage sidecar ... Lets the vocab maps, event-id split index, and normalizer stats be warmed once for a dataset (right after `dwarf convert`, or before a `dwarf hparam-scan` sweep) without needing to also start training. Extracts the setup-stage logic out of giant/pipeline.py:run_train_job into a standalone run_setup_stage() (returning a SetupStageResult), reused by both run_train_job and the new dwarf command's scripts/warm_setup_cache.py — a behavior-preserving refactor, covered by the existing test_pipeline.py suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:53:31 +02:00
lars	26aa9d3fde	Pass --seed through to the train/val event split ... make_event_split() defaults to seed=42, and run_train_job was calling it without forwarding t["seed"] — so the configured --seed affected model init/EMA/etc. but not which events landed in train vs. val, which silently always used seed 42 regardless of --seed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:29:19 +02:00
lars	e7478c36fb	Cache giant train's setup stage in a sidecar file ... Building the pdg/material vocab maps, the process map, and fitting the Stage-1/Stage-2 normalizers all require scanning the training dataset before a single epoch runs, which is wasted work whenever the same data path is reused across runs (hyperparameter sweeps via `dwarf hparam-scan`, repeated manual training attempts, ...). Persist those setup-stage outputs to a JSON sidecar next to the input data (giant/data/setup_cache.py), validated by a file fingerprint plus fixed dimension constants and a manually-bumped format version before reuse, with a soft warning (not a hard invalidation) on a git-hash mismatch alone. Also derives n_train_steps instantly from cached per-event row counts instead of accumulating it during the normalizer scan, and always collects the energy-router reservoir sample while the cache is being populated (not only when the current run's router is energy-typed) so a later run enabling --router-type energy never needs to rescan just to seed expert centers. New --cache-setup/--no-cache-setup (default on) and --rebuild-setup-cache/--no-rebuild-setup-cache flags on `giant train`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 10:28:37 +02:00
lars	9112625a08	Merge pull request 'Speed up giant train's setup stage' (#20) from perf/setup-stage into master ... Reviewed-on: #20	2026-07-29 14:02:27 +02:00
lars	84efbf5c2c	Merge branch 'master' into perf/setup-stage	2026-07-29 13:56:59 +02:00
lars	759b67a9e1	Speed up _WelfordAccumulator's per-chunk update ... The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:55:07 +02:00
lars	47a6c9db1f	Add regression coverage for vocab/process index-map builders ... build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:45:33 +02:00
lars	e331148afa	Merge pull request 'Make default checkpoint out_dir name reflect only non-default hyperparams' (#19) from feature/smart-outdir-naming into master ... Reviewed-on: #19	2026-07-29 13:38:16 +02:00
lars	09e4c765c7	Speed up giant train's setup stage ... Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:34:32 +02:00
lars	1115451c8e	Make default checkpoint out_dir name reflect only non-default hyperparams ... Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs.	2026-07-29 11:21:02 +02:00
lars	f2f89023d5	Merge pull request 'Feature/wandb integration' (#18) from feature/wandb-integration into master ... Reviewed-on: #18	2026-07-29 10:52:52 +02:00
lars	4c19072724	Skip empty-slice mean/std in sec phys validation print ... The per-dim print loop lacked the empty-array guard already used for the KL computation right above it and the sec-slot loop further down, so an all-zero-secondaries validation batch (e.g. early/unstable training) triggered numpy RuntimeWarnings from .mean()/.std() on empty arrays. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:49:12 +02:00
lars	539b6f61e1	Add test coverage for resolve_expert_dims ... Covers the default-config 0-sentinel inheritance path (the exact bug fixed by 969c5c6, previously untested since every router test in test_router.py passes expert_hidden_dim/expert_n_blocks explicitly), plus missing-key inheritance, full override, and partial override. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:44 +02:00
lars	f427d3384f	Timestamp default checkpoint dir to avoid W&B run-id collisions ... out_dir (and thus the W&B run id, which is derived from out_dir.name) was previously date-only, so two fresh runs on the same day with identical hyperparams silently shared one W&B run history. Default out_dir is now timestamped to the second. --resume without an explicit --out now reuses the checkpoint's own parent directory instead of recomputing a hyperparam-derived name, which both preserves the old continue-in-place behavior and fixes a latent bug where a resumed run with a changed hyperparam (e.g. --lr) would silently start writing to a new directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:40 +02:00
lars	bb699d41b2	Persist global_step across --resume so W&B step stays monotonic ... Previously global_step always reset to 0, even on --resume. Since the W&B run reattaches to the same run id on resume, logging with step=global_step after a restart passed step values below what was already recorded, silently dropping the resumed portion's metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:35 +02:00
lars	a986f96ba3	Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B ... Adds Router.gate_stats (per-router gate entropy + per-expert utilization), logged both per-batch (entropy only, train loop) and per-epoch (full stats, over the whole val set) — the router-collapse failure mode from the roadmap's rollout postmortem is now visible during training instead of only after a full rollout+analysis run. Also splits WGAN critic/ generator grad norms instead of summing them, logs critic LR, n_sec head accuracy, GPU peak memory + samples/sec, model parameter counts (in wandb.config), and an is_best flag — all wired into both metrics.csv and W&B. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:21:03 +02:00
lars	969c5c6e9a	Fix router experts silently ignoring --hidden-dim/--n-blocks ... expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed run always got fixed 128/3-wide experts no matter what --hidden-dim/ --n-blocks was passed. They now default to 0 ("unset"), which resolve_expert_dims() resolves by inheriting the model dims; an explicit override still works and now warns when it diverges from model.hidden_dim/n_blocks, since the checkpoint dir name won't reflect it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:34:25 +02:00
lars	29459ab1f7	Log batch-level metrics to W&B, not just per-epoch summaries ... giant train --wandb now also logs loss/grad_norm/lr every N optimizer steps (--wandb-log-every, default 50) so W&B shows within-epoch trends, not just one point per epoch. Both share global_step as a single monotonic step axis (wandb.Run.log requires step to never decrease across calls), which also fixes global_step previously only advancing in wgan mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:20:05 +02:00
lars	a05837f918	Apply ruff format ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:11:05 +02:00
lars	0778a61360	Add opt-in Weights & Biases logging for the training loop ... giant train --wandb logs the same per-epoch metrics already written to metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`) that nothing else depends on. A run's id is derived from the checkpoint out_dir so --resume reattaches to the existing run instead of starting a new one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:07:30 +02:00
lars	db0f12be58	Add configs for router energy (embedding/physical) and WGAN baseline runs ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 15:53:04 +02:00
lars	6208810089	Merge pull request 'Refactor the analysis plot creation with focus on rollout' (#16) from analysis-rollout-plots into master ... Reviewed-on: #16	2026-07-27 14:58:51 +02:00
lars	5594866eac	Merge remote-tracking branch 'origin/master' into analysis-rollout-plots ... # Conflicts: # analysis/rollout_validation.ipynb	2026-07-27 14:55:00 +02:00
lars	0f7febfd02	ci: set UV_LINK_MODE=copy to silence the cross-filesystem hardlink warning ... The bind-mounted uv cache and the job workspace are on different filesystems, so uv already falls back to copying installed files; this just tells it to do so directly instead of logging a hardlink-failed warning every job.	2026-07-27 14:40:01 +02:00
lars	19be455346	ci: re-pin UV_CACHE_DIR after setup-uv, which exports its own value regardless of enable-cache ... Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path (/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step right after setup-uv so it wins for the rest of the job.	2026-07-27 14:31:22 +02:00
lars	1787d26d1c	ci: stop setup-uv from overriding UV_CACHE_DIR ... setup-uv's default enable-cache: auto sets its own UV_CACHE_DIR (a tool-cache tmp path) as a later step, clobbering the workflow-level UV_CACHE_DIR that points at the bind-mounted persistent cache. Disable setup-uv's own cache handling so our mount stays in effect.	2026-07-27 14:22:49 +02:00
lars	0afa75ee30	ci: replace unsupported artifact sharing with a bind-mounted uv cache ... The self-hosted act_runner doesn't support upload/download-artifact, so drop that plumbing and instead run jobs in an explicit container with a persistent host directory mounted at /uv-cache (UV_CACHE_DIR), backed by valid_volumes on the runner. uv sync still runs per job but hits a warm local cache instead of re-downloading/building packages every time.	2026-07-27 14:18:54 +02:00
lars	08c76a9614	ci: share one uv sync across jobs, gate tests on lint+type-check, sync tag/version on release tags ... Uploads the synced .venv as an artifact from a single setup job instead of re-running uv sync (and re-downloading torch) in every job. Drops the build/publish job in favor of a lighter job that, on a pushed tag, checks the tag against the uv project version and — if they differ — bumps the version, commits it to master, and recreates the tag on the new commit.	2026-07-27 14:00:59 +02:00
lars	b8a4dc7d63	analyze: thread full model/training/rollout/dataset params to plots ... giant rollout now writes the checkpoint's complete model_config (incl. the router sub-dict), the sibling config.toml's [train]/[meta] sections, and every rollout CLI knob (weights, batch_size, escape_threshold, n_events, device, seed) into the YAML sidecar instead of a hand-picked subset. All of it flows through run_meta.json into each plot's own metadata.yaml for later comparison, while the figure subtitle itself shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router, epoch, best_val_loss, steps/noise_dim) via new_figure's params option. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 13:35:02 +02:00
lars	4feb9007f3	docs: note the EnergyRouter centers_init fix in the roadmap ... Records both identified contributors to the 2026-07-22 router divergence (missing lambda_balance and the uniform-linspace center init) and that the latter is now fixed, with next steps covering both for the retrain.	2026-07-27 13:14:49 +02:00
lars	ee29b9a303	router: seed EnergyRouter centers from data quantiles instead of a fixed linspace ... The 2026-07-22 rollout benchmark's router_gating diagnostic showed the 10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a roughly uniform z-normalized energy distribution, leaving experts heavily overlapping instead of partitioning the range. Add an optional centers_init kwarg (backward compatible, defaults to the old linspace) and have giant train estimate it from a reservoir sample of the real energy column, collected during the existing normalizer-fitting pass.	2026-07-27 13:11:16 +02:00
lars	641bbb0a68	docs: record first MoE router rollout benchmark result in the roadmap ... The 10-expert EnergyRouter checkpoint (lambda_balance=0.0) diverges badly from Geant4 on rollout (step granularity, secondary species, shower shape), and the router_gating diagnostic shows the experts heavily overlap rather than partitioning the pre-step energy domain. Recorded as "needs retraining with a different router config" rather than an abandoned direction. Full analysis in the knowledge base at experiments/giant-router-energy-rollout-validation.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 12:02:59 +02:00
lars	61410ddee3	analyze: default run directory to <repo>/analysis_runs, gitignored ... giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.	2026-07-27 11:29:35 +02:00
lars	a88b21ef70	style: ruff format	2026-07-27 11:19:06 +02:00
lars	eb751d968d	transforms: pad legacy cond normalizers for pre-physical-conditioning checkpoints ... Checkpoints trained before commit 68fb99b (physical-property conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond normalizer, fit before build_cond_features grew the extra physical columns. Any inference against such a checkpoint under current code (predict/rollout/router_gating) crashed broadcasting a 15-wide cond_cont against an 8-wide mean/std. In "embedding" mode those physical columns are never read by ConditionEncoder, so padding the missing entries with mean=0/std=1 is a safe no-op. "physical" mode reads them directly, so a mismatch there still raises instead of silently normalizing garbage.	2026-07-27 11:06:48 +02:00
lars	4d6101dcd7	analyze: recalibrate condor walltime model from real cluster timings ... The prior _COST_MODEL/_FIXED_OVERHEAD_S were fit only against local synthetic benchmarks (up to 2M rows/side), which can't see docker pull or real /ceph read latency and wildly overestimated real jobs (~1200-1800s predicted vs 50-320s median actual, from condor_history on production run 563f5ee3, --chunks 4, ~254M total rows). Refit each spec's per-row rate through the origin against its median real wall-clock time (not max, to avoid baking a few /ceph-contention spikes into a rate that would then wrongly scale with dataset size), and raised RUNTIME_SAFETY_MARGIN to compensate for that same contention risk instead.	2026-07-27 11:06:40 +02:00
lars	c7194701f6	analyze: raise default condor job memory request to 8192 MB ... 4096 MB was too tight: a resubmitted run held ~31 jobs spread evenly across nearly every plot family and chunk index with "Docker job has gone over memory limit of 4224 Mb", not one specific spec, so the generic per-chunk data footprint needed more headroom.	2026-07-27 10:31:46 +02:00
lars	dbd5c7e083	analyze: default condor docker image to alma9-gridjob ... mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9 not found). Switch the default to cverstege/alma9-gridjob, a modern EL9-based image.	2026-07-27 10:14:15 +02:00
lars	fa59443339	analyze: run condor compute jobs via .venv/bin/giant, not uv run ... uv isn't installed on the HTCondor worker docker image, so `uv run` fails there. giant is already an installed console script in the repo's uv-synced .venv, so exec it directly instead. write_submit now fails fast with a clear message if .venv/bin/giant is missing.	2026-07-27 10:03:02 +02:00
lars	e380400fe9	analyze: estimate per-job HTCondor walltime from chunk row count ... Each condor job's +RequestWalltime used to be one flat 3600s default for every (plot, chunk), regardless of how much data it actually streams over. `prep` now records each chunk's rollout+reference row count, and `giant/analysis/runtime_estimate.py` turns that into a per-job estimate: a per-spec (intercept, seconds/row) cost model fit by `scripts/profile_analysis_costs.py` against synthetic mock data on this machine, plus a fixed overhead placeholder (docker/uv/shared-fs startup — unmeasurable here, no /ceph access) and a single RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and the submit description references it via $(walltime) instead of a constant.	2026-07-27 09:47:44 +02:00
lars	85d3914a4d	analyze: expose bin/pdg options on analyze submit ... `submit` calls `prep` internally but only forwarded --chunks, so a condor run could never use non-default energy-bins/bins/top-pdg.	2026-07-27 09:30:54 +02:00
lars	86fc46b5a8	analyze: chunk per-plot aggregation across HTCondor jobs ... Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.	2026-07-27 09:24:19 +02:00
lars	70d0f04326	analyze: normalize pdg dtype in open_side to fix rollout/reference concat ... Rollout output and the reference file's ROOT-derived parquet disagree on pdg's integer width (Int32 vs Int64), which only surfaced downstream as a pl.concat SchemaError in build_context's pdg-count merge. Cast to a canonical Int64 at the single scan entry point instead.	2026-07-24 14:32:06 +02:00
lars	80d544aa73	docs: document compute environment, WGAN/MoE status, and condor-gpu-train-rollout ... Adds a Compute environment section (laptop/desktop vs. shared portal machines vs. condor workers) and corrects the roadmap: the WGAN-GP and MoE routing-trunk tracks are actually implemented (untested and under-testing respectively), not "not yet built" as previously stated. Also notes the in-progress condor-gpu-train-rollout branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 14:20:14 +02:00
lars	b6893b0118	chore: remove stray CUDA sanity script and stale Phase 2 planning doc ... test-cuda.py was a one-off local CUDA check, not part of tests/ or scripts/. docs/phase2_plan.md is superseded by the "Phase 2 (implemented)" section of CLAUDE.md's roadmap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:49:00 +02:00
lars	60c2ca1985	analyze: add MoE router gating/share diagnostic plots ... New "model" family in the gallery: router_gating (mean soft gate weight vs. pre-step energy, showing the router's soft decision boundaries) and router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share by species / true physics process). Needs a live checkpoint's Router, so it's a documented exception to the rest of the package's polars/numpy-only contract; gracefully degrades to a placeholder for non-MoE checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:32:25 +02:00
lars	1115301eb9	test: replace prep(**_CTX) splat with a typed _prep helper ... ty correctly flagged this as unsound: _CTX's inferred dict[str, int] type doesn't rule out a "run_dir" key, which would silently bind to prep's own run_dir: str | Path | None parameter instead of falling through to **ctx_kwargs. Passing the context kwargs by name in a small test helper removes the ambiguity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 11:17:45 +02:00
lars	77b2e9e5f4	analyze: drop stale ty:ignore on plotstyle import ... plotstyle is now a real installed package (from the git.larsbogner.de index) with its own type info, so the unresolved-import suppression is no longer needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 11:17:41 +02:00