169 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
lars	9112625a08	Merge pull request 'Speed up giant train's setup stage' (#20) from perf/setup-stage into master ... Reviewed-on: #20	2026-07-29 14:02:27 +02:00
lars	84efbf5c2c	Merge branch 'master' into perf/setup-stage	2026-07-29 13:56:59 +02:00
lars	759b67a9e1	Speed up _WelfordAccumulator's per-chunk update ... The streaming update re-derived two full (B, F) arrays from the running mean (once before updating it, once after) plus an elementwise product — five passes over each chunk and three temporary arrays, to maintain a mean/variance that's tiny in width (COND_DIM=15 at most). Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm: compute the chunk's own local mean/M2 (independent of the running state) and merge it in with an O(F) combination formula. Same streaming interface and output (identical to ~1e-14, float64 rounding noise), ~40% faster per update() call on a benchmark chunk. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:55:07 +02:00
lars	47a6c9db1f	Add regression coverage for vocab/process index-map builders ... build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:45:33 +02:00
lars	e331148afa	Merge pull request 'Make default checkpoint out_dir name reflect only non-default hyperparams' (#19) from feature/smart-outdir-naming into master ... Reviewed-on: #19	2026-07-29 13:38:16 +02:00
lars	09e4c765c7	Speed up giant train's setup stage ... Fits normalizers over multi-hundred-million-row datasets, so the setup pass's per-row Python overhead compounds fast: encode_secondaries recomputed an O(K) prefix sum from scratch on every one of its 15 stick-breaking iterations, np.isin re-sorted the full train-event-id array on every chunk, and pdg/material/process index lookups ran a Python dict lookup per row. The normalizer-fit pass also computed encode_secondaries's stick-logit and direction-rotation blocks in full even though it only ever reads the mass/charge columns. Replace the prefix-sum recompute with a single np.cumsum, add a sorted_membership helper (searchsorted-based) in place of np.isin at both the setup-pass and per-epoch call sites, vectorize the index lookups via _vectorized_map_lookup, and add an opt-in phys_only path so the setup pass skips the stick-breaking/rotation work it discards anyway. All four changes are output-identical performance refactors, backed by new unit tests plus the existing suite. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:34:32 +02:00
lars	1115451c8e	Make default checkpoint out_dir name reflect only non-default hyperparams ... Previously the same fixed 7 fields (mode/hidden_dim/n_blocks/emb_dim/ conditioning/lr/batch_size) were always baked into the name, even for a vanilla run, and router config wasn't represented at all. Now default_out_dir_name only includes fields that differ from DEFAULT_CONFIG, adds router/seed/epochs as candidates, and caps at 6 shown fields with a hashed overflow suffix for heavily-swept configs.	2026-07-29 11:21:02 +02:00
lars	f2f89023d5	Merge pull request 'Feature/wandb integration' (#18) from feature/wandb-integration into master ... Reviewed-on: #18	2026-07-29 10:52:52 +02:00
lars	4c19072724	Skip empty-slice mean/std in sec phys validation print ... The per-dim print loop lacked the empty-array guard already used for the KL computation right above it and the sec-slot loop further down, so an all-zero-secondaries validation batch (e.g. early/unstable training) triggered numpy RuntimeWarnings from .mean()/.std() on empty arrays. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:49:12 +02:00
lars	539b6f61e1	Add test coverage for resolve_expert_dims ... Covers the default-config 0-sentinel inheritance path (the exact bug fixed by 969c5c6, previously untested since every router test in test_router.py passes expert_hidden_dim/expert_n_blocks explicitly), plus missing-key inheritance, full override, and partial override. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:44 +02:00
lars	f427d3384f	Timestamp default checkpoint dir to avoid W&B run-id collisions ... out_dir (and thus the W&B run id, which is derived from out_dir.name) was previously date-only, so two fresh runs on the same day with identical hyperparams silently shared one W&B run history. Default out_dir is now timestamped to the second. --resume without an explicit --out now reuses the checkpoint's own parent directory instead of recomputing a hyperparam-derived name, which both preserves the old continue-in-place behavior and fixes a latent bug where a resumed run with a changed hyperparam (e.g. --lr) would silently start writing to a new directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:40 +02:00
lars	bb699d41b2	Persist global_step across --resume so W&B step stays monotonic ... Previously global_step always reset to 0, even on --resume. Since the W&B run reattaches to the same run id on resume, logging with step=global_step after a restart passed step values below what was already recorded, silently dropping the resumed portion's metrics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:42:35 +02:00
lars	a986f96ba3	Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B ... Adds Router.gate_stats (per-router gate entropy + per-expert utilization), logged both per-batch (entropy only, train loop) and per-epoch (full stats, over the whole val set) — the router-collapse failure mode from the roadmap's rollout postmortem is now visible during training instead of only after a full rollout+analysis run. Also splits WGAN critic/ generator grad norms instead of summing them, logs critic LR, n_sec head accuracy, GPU peak memory + samples/sec, model parameter counts (in wandb.config), and an is_best flag — all wired into both metrics.csv and W&B. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 10:21:03 +02:00
lars	969c5c6e9a	Fix router experts silently ignoring --hidden-dim/--n-blocks ... expert_hidden_dim/expert_n_blocks were hardcoded to 128/3 in DEFAULT_CONFIG, independent of model.hidden_dim/n_blocks, so a routed run always got fixed 128/3-wide experts no matter what --hidden-dim/ --n-blocks was passed. They now default to 0 ("unset"), which resolve_expert_dims() resolves by inheriting the model dims; an explicit override still works and now warns when it diverges from model.hidden_dim/n_blocks, since the checkpoint dir name won't reflect it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:34:25 +02:00
lars	29459ab1f7	Log batch-level metrics to W&B, not just per-epoch summaries ... giant train --wandb now also logs loss/grad_norm/lr every N optimizer steps (--wandb-log-every, default 50) so W&B shows within-epoch trends, not just one point per epoch. Both share global_step as a single monotonic step axis (wandb.Run.log requires step to never decrease across calls), which also fixes global_step previously only advancing in wgan mode. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:20:05 +02:00
lars	a05837f918	Apply ruff format ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:11:05 +02:00
lars	0778a61360	Add opt-in Weights & Biases logging for the training loop ... giant train --wandb logs the same per-epoch metrics already written to metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`) that nothing else depends on. A run's id is derived from the checkpoint out_dir so --resume reattaches to the existing run instead of starting a new one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 16:07:30 +02:00
lars	db0f12be58	Add configs for router energy (embedding/physical) and WGAN baseline runs ... Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 15:53:04 +02:00
lars	6208810089	Merge pull request 'Refactor the analysis plot creation with focus on rollout' (#16) from analysis-rollout-plots into master ... Reviewed-on: #16	2026-07-27 14:58:51 +02:00
lars	5594866eac	Merge remote-tracking branch 'origin/master' into analysis-rollout-plots ... # Conflicts: # analysis/rollout_validation.ipynb	2026-07-27 14:55:00 +02:00
lars	0f7febfd02	ci: set UV_LINK_MODE=copy to silence the cross-filesystem hardlink warning ... The bind-mounted uv cache and the job workspace are on different filesystems, so uv already falls back to copying installed files; this just tells it to do so directly instead of logging a hardlink-failed warning every job.	2026-07-27 14:40:01 +02:00
lars	19be455346	ci: re-pin UV_CACHE_DIR after setup-uv, which exports its own value regardless of enable-cache ... Log evidence showed setup-uv still sets UV_CACHE_DIR to a tool-cache tmp path (/tmp/setup-uv-cache) even with enable-cache: false, clobbering the workflow env pointing at the bind-mounted cache. Re-export it via GITHUB_ENV in a step right after setup-uv so it wins for the rest of the job.	2026-07-27 14:31:22 +02:00
lars	1787d26d1c	ci: stop setup-uv from overriding UV_CACHE_DIR ... setup-uv's default enable-cache: auto sets its own UV_CACHE_DIR (a tool-cache tmp path) as a later step, clobbering the workflow-level UV_CACHE_DIR that points at the bind-mounted persistent cache. Disable setup-uv's own cache handling so our mount stays in effect.	2026-07-27 14:22:49 +02:00
lars	0afa75ee30	ci: replace unsupported artifact sharing with a bind-mounted uv cache ... The self-hosted act_runner doesn't support upload/download-artifact, so drop that plumbing and instead run jobs in an explicit container with a persistent host directory mounted at /uv-cache (UV_CACHE_DIR), backed by valid_volumes on the runner. uv sync still runs per job but hits a warm local cache instead of re-downloading/building packages every time.	2026-07-27 14:18:54 +02:00
lars	08c76a9614	ci: share one uv sync across jobs, gate tests on lint+type-check, sync tag/version on release tags ... Uploads the synced .venv as an artifact from a single setup job instead of re-running uv sync (and re-downloading torch) in every job. Drops the build/publish job in favor of a lighter job that, on a pushed tag, checks the tag against the uv project version and — if they differ — bumps the version, commits it to master, and recreates the tag on the new commit.	2026-07-27 14:00:59 +02:00
lars	b8a4dc7d63	analyze: thread full model/training/rollout/dataset params to plots ... giant rollout now writes the checkpoint's complete model_config (incl. the router sub-dict), the sibling config.toml's [train]/[meta] sections, and every rollout CLI knob (weights, batch_size, escape_threshold, n_events, device, seed) into the YAML sidecar instead of a hand-picked subset. All of it flows through run_meta.json into each plot's own metadata.yaml for later comparison, while the figure subtitle itself shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router, epoch, best_val_loss, steps/noise_dim) via new_figure's params option. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 13:35:02 +02:00
lars	4feb9007f3	docs: note the EnergyRouter centers_init fix in the roadmap ... Records both identified contributors to the 2026-07-22 router divergence (missing lambda_balance and the uniform-linspace center init) and that the latter is now fixed, with next steps covering both for the retrain.	2026-07-27 13:14:49 +02:00
lars	ee29b9a303	router: seed EnergyRouter centers from data quantiles instead of a fixed linspace ... The 2026-07-22 rollout benchmark's router_gating diagnostic showed the 10-expert EnergyRouter's default linspace(-2, 2, n_experts) init assumes a roughly uniform z-normalized energy distribution, leaving experts heavily overlapping instead of partitioning the range. Add an optional centers_init kwarg (backward compatible, defaults to the old linspace) and have giant train estimate it from a reservoir sample of the real energy column, collected during the existing normalizer-fitting pass.	2026-07-27 13:11:16 +02:00
lars	641bbb0a68	docs: record first MoE router rollout benchmark result in the roadmap ... The 10-expert EnergyRouter checkpoint (lambda_balance=0.0) diverges badly from Geant4 on rollout (step granularity, secondary species, shower shape), and the router_gating diagnostic shows the experts heavily overlap rather than partitioning the pre-step energy domain. Recorded as "needs retraining with a different router config" rather than an abandoned direction. Full analysis in the knowledge base at experiments/giant-router-energy-rollout-validation.md. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 12:02:59 +02:00
lars	61410ddee3	analyze: default run directory to <repo>/analysis_runs, gitignored ... giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.	2026-07-27 11:29:35 +02:00
lars	a88b21ef70	style: ruff format	2026-07-27 11:19:06 +02:00
lars	eb751d968d	transforms: pad legacy cond normalizers for pre-physical-conditioning checkpoints ... Checkpoints trained before commit 68fb99b (physical-property conditioning, COND_DIM 8->15) saved a COND_DIM_BASE-wide cond normalizer, fit before build_cond_features grew the extra physical columns. Any inference against such a checkpoint under current code (predict/rollout/router_gating) crashed broadcasting a 15-wide cond_cont against an 8-wide mean/std. In "embedding" mode those physical columns are never read by ConditionEncoder, so padding the missing entries with mean=0/std=1 is a safe no-op. "physical" mode reads them directly, so a mismatch there still raises instead of silently normalizing garbage.	2026-07-27 11:06:48 +02:00
lars	4d6101dcd7	analyze: recalibrate condor walltime model from real cluster timings ... The prior _COST_MODEL/_FIXED_OVERHEAD_S were fit only against local synthetic benchmarks (up to 2M rows/side), which can't see docker pull or real /ceph read latency and wildly overestimated real jobs (~1200-1800s predicted vs 50-320s median actual, from condor_history on production run 563f5ee3, --chunks 4, ~254M total rows). Refit each spec's per-row rate through the origin against its median real wall-clock time (not max, to avoid baking a few /ceph-contention spikes into a rate that would then wrongly scale with dataset size), and raised RUNTIME_SAFETY_MARGIN to compensate for that same contention risk instead.	2026-07-27 11:06:40 +02:00
lars	c7194701f6	analyze: raise default condor job memory request to 8192 MB ... 4096 MB was too tight: a resubmitted run held ~31 jobs spread evenly across nearly every plot family and chunk index with "Docker job has gone over memory limit of 4224 Mb", not one specific spec, so the generic per-chunk data footprint needed more headroom.	2026-07-27 10:31:46 +02:00
lars	dbd5c7e083	analyze: default condor docker image to alma9-gridjob ... mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9 not found). Switch the default to cverstege/alma9-gridjob, a modern EL9-based image.	2026-07-27 10:14:15 +02:00
lars	fa59443339	analyze: run condor compute jobs via .venv/bin/giant, not uv run ... uv isn't installed on the HTCondor worker docker image, so `uv run` fails there. giant is already an installed console script in the repo's uv-synced .venv, so exec it directly instead. write_submit now fails fast with a clear message if .venv/bin/giant is missing.	2026-07-27 10:03:02 +02:00
lars	e380400fe9	analyze: estimate per-job HTCondor walltime from chunk row count ... Each condor job's +RequestWalltime used to be one flat 3600s default for every (plot, chunk), regardless of how much data it actually streams over. `prep` now records each chunk's rollout+reference row count, and `giant/analysis/runtime_estimate.py` turns that into a per-job estimate: a per-spec (intercept, seconds/row) cost model fit by `scripts/profile_analysis_costs.py` against synthetic mock data on this machine, plus a fixed overhead placeholder (docker/uv/shared-fs startup — unmeasurable here, no /ceph access) and a single RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and the submit description references it via $(walltime) instead of a constant.	2026-07-27 09:47:44 +02:00
lars	85d3914a4d	analyze: expose bin/pdg options on analyze submit ... `submit` calls `prep` internally but only forwarded --chunks, so a condor run could never use non-default energy-bins/bins/top-pdg.	2026-07-27 09:30:54 +02:00
lars	86fc46b5a8	analyze: chunk per-plot aggregation across HTCondor jobs ... Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.	2026-07-27 09:24:19 +02:00
lars	70d0f04326	analyze: normalize pdg dtype in open_side to fix rollout/reference concat ... Rollout output and the reference file's ROOT-derived parquet disagree on pdg's integer width (Int32 vs Int64), which only surfaced downstream as a pl.concat SchemaError in build_context's pdg-count merge. Cast to a canonical Int64 at the single scan entry point instead.	2026-07-24 14:32:06 +02:00
lars	80d544aa73	docs: document compute environment, WGAN/MoE status, and condor-gpu-train-rollout ... Adds a Compute environment section (laptop/desktop vs. shared portal machines vs. condor workers) and corrects the roadmap: the WGAN-GP and MoE routing-trunk tracks are actually implemented (untested and under-testing respectively), not "not yet built" as previously stated. Also notes the in-progress condor-gpu-train-rollout branch. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 14:20:14 +02:00
lars	b6893b0118	chore: remove stray CUDA sanity script and stale Phase 2 planning doc ... test-cuda.py was a one-off local CUDA check, not part of tests/ or scripts/. docs/phase2_plan.md is superseded by the "Phase 2 (implemented)" section of CLAUDE.md's roadmap. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:49:00 +02:00
lars	60c2ca1985	analyze: add MoE router gating/share diagnostic plots ... New "model" family in the gallery: router_gating (mean soft gate weight vs. pre-step energy, showing the router's soft decision boundaries) and router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share by species / true physics process). Needs a live checkpoint's Router, so it's a documented exception to the rest of the package's polars/numpy-only contract; gracefully degrades to a placeholder for non-MoE checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:32:25 +02:00
lars	1115301eb9	test: replace prep(**_CTX) splat with a typed _prep helper ... ty correctly flagged this as unsound: _CTX's inferred dict[str, int] type doesn't rule out a "run_dir" key, which would silently bind to prep's own run_dir: str | Path | None parameter instead of falling through to **ctx_kwargs. Passing the context kwargs by name in a small test helper removes the ambiguity. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 11:17:45 +02:00
lars	77b2e9e5f4	analyze: drop stale ty:ignore on plotstyle import ... plotstyle is now a real installed package (from the git.larsbogner.de index) with its own type info, so the unresolved-import suppression is no longer needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 11:17:41 +02:00
lars	6e1d11b8d8	deps: install plotstyle from git.larsbogner.de package index ... Swap the local/editable ETPlot `gallery[plotting]` dependency for `plotstyle>=1.0.0`, now published to a package registry, so the repo doesn't need a local ETPlot checkout to resolve. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 11:11:31 +02:00
lars	b6ed618517	Run rollout validation for the mixture of experts at 50 GeV.	2026-07-24 09:59:30 +02:00
lars	1e7d7d7efd	analyze: show model/training params on rendered figures ... `giant rollout` now records the checkpoint's architecture (mode, hidden_dim, n_blocks, emb_dim, dropout, conditioning) plus training_epoch and best_val_loss in its YAML sidecar, using data already loaded from the checkpoint. condor.py carries those through run_meta, and render.py passes them to plotstyle's new_figure(params=...) so every plot's subtitle shows what produced it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 09:54:21 +02:00
lars	8ff70e3c87	analyze: drive prep/submit from the rollout YAML sidecar ... `giant analyze prep` / `submit` now take the `giant rollout` YAML sidecar as their only positional input instead of explicit --rollout/--reference/--out-dir. The YAML's `output`/`dataset` keys name the rollout parquet and its seed file (the reference truth), and the rest of the sidecar (checkpoint, geometry oracle, cutoffs) flows into every plot's gallery metadata. prep derives its own run directory next to the rollout parquet (<...>/analysis_<id>/) holding shared.json, run_meta.json, reduced/, plots/. compute-one and render now take just --run-dir / a run-dir argument and read the resolved paths + metadata from run_meta.json, so the condor wrapper no longer threads file paths. open_side scans a directory of reference shards via glob. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:56:36 +02:00
lars	f4c2545e8b	Rewrite analysis as streaming rollout-vs-reference plotting pipeline ... Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth diagnostics) with a lean giant/analysis/ package that compares one autoregressive `giant rollout` for a checkpoint against a held-out miniCaloSim reference file, and generates publication-styled plots in parallel on HTCondor. Rollout output and a raw reference file share a world-frame physical column subset under identical names, so the old ALR/local-frame decode machinery is gone — everything is world-frame mm/MeV. - sources.py: canonical LazyFrames, synthetic-termination-row filtering, the secondary view (rollout generation>0 tracks vs reference sec_*_list). - reduce.py: streaming primitives — a single hist1d group_by pass, per-event scalars, edep-weighted depth/transverse profiles, species share, leakage. - context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material group sets once into shared.json, so each compute job is one pass, no range scan (histogram efficiency). - catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg, material}, per-event totals, shower profiles, species/leakage, secondaries. - render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata. - condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit): one job per plot, compute/render split (workers polars-only, no LaTeX). Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover the reduce primitives, catalog id uniqueness + compute, condor submit, and a guarded render smoke test. Delete the two predict-diagnostics notebooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:38:09 +02:00