24 Commits

Author	SHA1	Message	Date
lars	81eb14d75c	Move scripts/ to giant/tools/ (issues.md Issue 9) ... `scripts` was published as a top-level distribution package, colliding with one of the most generic names in the Python ecosystem and shadowable by a stray scripts/ dir on the portal machines' shared /work/lbogner. Move it under the giant namespace; the dwarf command name is unchanged, only the Python import path and file location move. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-13 10:31:47 +02:00
lars	818c380fd0	Extract predict/rollout's duplicated inference bootstrap into giant.checkpoint_io (issues.md Issue 5) ... giant predict and giant rollout each carried a ~65-line, independently drifting copy of "load checkpoint -> validate -> resolve conditioning axes -> restore normalizers/vocab maps -> build models -> load weights", plus a third partial copy of _conditioning_axes in analysis/router_gating.py. A silent divergence there doesn't crash, it makes the two commands run different physics from the same checkpoint with no test coverage anywhere along that path. giant/checkpoint_io.py now holds the single implementation: load_for_inference() + an InferenceContext dataclass, raising CheckpointCompatibilityError (verbatim message text preserved) instead of calling typer directly, so it can be unit-tested and imported from non-Typer code. router_gating.py's load_router imports conditioning_axes from it lazily, keeping its "no torch at module scope" contract intact. Adds 17 direct unit tests for load_for_inference/conditioning_axes/stage_cfg plus CLI smoke tests confirming the error surfaces as typer.Exit(1) through predict and rollout — previously zero coverage on this path. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 15:31:14 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	878e9ddca3	Delete docs/v0.3.0-design.md and strip all references to it ... The design doc and its followups doc are no longer needed as a live reference now that the v0.3.0 redesign is implemented — comments and docstrings across the codebase cited it extensively (file path, "design doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design rationale. Removed docs/ and edited every citing comment/docstring to drop the now-dangling reference while keeping the substantive explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its trailing pointer to the deleted file. Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or "§N.N" references (repo-wide grep); ruff and ty clean; full test suite on the heaviest-touched modules (network, sample, rollout, migration, config, train) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:19:02 +02:00
lars	da7cde3ef9	v0.3.0 post-implementation audit: resolve all 9 tracked discrepancies ... Works through docs/v0.3.0-followups.md item by item, closing the gap between the design doc and the shipped v0.3.0-stage2-autoregressive code: 1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2 dispatch, stage-2 particle-type-class marginal. 2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run. 3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/ train instead of the hardcoded K_MAX constant. 4. Mixed conditioning.particle.type / conditioning.material.type support end-to-end (data pipeline + dwarf warm-cache). 5. conditioning.share_stages = true: one shared ConditionEncoder instance across both stages. 6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2 (was silently unimplemented). 7. giant predict/rollout: implement conditioning.*.type = "onehot" via the checkpoint's saved pdg_topn_map/mat_topn_map. 8. network.py's checkpoint-path model_config migration now fails loudly on non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's TOML-load path (§4.2). 9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a rollout-capable checkpoint (§9). Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics), mostly a test-helper dict-unpack pattern that made every unrelated constructor keyword look like a type error. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 16:12:58 +02:00
lars	93b19911f8	v0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode ... - giant/sample.py: fix every sampler's call convention against Stage1Model/Stage2OneShot's actual forward signatures (was still calling model(x, t, cond_cont, cond_cat) positionally); add sample_secondaries_ar (free-running AR loop, unsnapped history feature) and sample_stage1/sample_stage2/resolve_n_sec dispatch helpers that read each stage's generator_kind/decoder off the model instance itself. - giant/particles.py: decode_topn_class (argmax + other_policy) and decode_embedding_nearest (L1-snap + distance) turn a secondary's "onehot"/"embedding" type prediction into a concrete PDG. - giant/rollout.py: decode_secondary_identity routes all three particle_type.target values to real mass/charge; per-stage generator dispatch (drops the single shared `mode` string, adds ddpm support); L1DistCollector accumulates the §11.3 embedding-distance diagnostic. - giant/cli.py: drop the onehot/embedding-target rejection gate (narrowed to the still-unimplemented conditioning.particle/material.type=onehot axis); fix the dead model_cfg.get("mode") bug in predict/rollout. - giant/analysis/: new type_embedding_l1_distance PlotSpec, wired through the rollout YAML sidecar (no live-model call needed, unlike router_gating -- the histogram is already pre-aggregated at rollout time). - Un-xfail every test that was blocked on this step (test_rollout.py, test_flow.py, test_wgan.py, test_phase2.py, test_router.py, test_validate.py); add test_sample.py, test_type_embedding_distance.py. Known follow-up: giant/validate.py still unpacks the training val-batch as a stale 6-tuple and doesn't use the new per-stage dispatch, so marginal validation during training degrades gracefully with a warning rather than working -- not in this step's scope. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-07 10:37:57 +02:00
lars	9112e845e0	v0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite ... Replaces train.py's single global training loop with a StageTrainer hierarchy (FlowDDPMStageTrainer, WGANStageTrainer) — one per active stage, each owning its own optimizer/LR schedule/EMA and reading only the shared batch tuple (stage 2 always teacher-forces on the ground-truth x1_s1, so stages never need each other's output at train time). Supports every stage1/stage2 generator combination, including the design doc's headline mixed case (stage1=flow + stage2=wgan) and its reverse, plus stage1-only/stage2-only ablation runs, routed+gumbel stages, and checkpoint save/resume. metrics.csv/wandb logging are stage-prefixed. validate_marginals calls are guarded with a one-time warning and a Wasserstein-magnitude fallback for wgan best-checkpoint selection, since giant/sample.py still assumes stage1 always owns n_sec_head (decision 1 moved it to stage 2 by default) — deferred to design doc step 6, not silently papered over. pipeline.py's run_setup_stage/run_train_job now read the new nested config directly; the dangling resolve_expert_dims call and the --mode wgan --router rejection are both gone (routed WGAN works). cli.py's train/new-run build correctly-shaped config overrides (architecture flags -> stage1_model only per the approved decision; --mode/--n-critic/--gp-weight/--critic-lr broadcast to both stages, matching migrate_config's own precedent and avoiding a regression on the common --mode case); predict/rollout's dangling build_models tuple-unpack is fixed; new-run now tags config_version, fixing a bug where a re-loaded v0.3 config.toml would have been silently corrupted by migrate_config mistaking it for v0.2. config.py's validate_config rejects mixed particle/material conditioning types for now (ConditionEncoder supports it, the data pipeline in giant/data/transforms.py doesn't yet). analysis/render.py and router_gating.py handle both the new nested model_config shape and legacy flat checkpoints. scripts/warm_setup_cache.py updated for run_setup_stage's new signature. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 11:31:49 +02:00
lars	c3e5956718	Resolve giant condor wrapper from the active venv, not a hardcoded path ... write_submit baked in cfg.repo_dir/.venv/bin/giant unconditionally, which breaks when submitting from a differently-named or non-default venv (e.g. --extra cuda). Prefer the giant executable next to sys.executable (the venv actually running the submit), falling back to repo_dir/.venv/bin/giant.	2026-08-03 14:46:54 +02:00
lars	ad1b8e7835	Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline ... - prep() now clears reduced_partial/ and reduced/ on every (re-)run. Partial files carry no record of what context (n_chunks, bin edges, group sets) they were computed under, so re-prepping the same run_dir with a different --chunks/--bins/--top-pdg (or after the rollout was regenerated) previously left old partials on disk that merge_one would silently merge against the new shared.json — producing a wrong-but-plausible reduced/*.json with no error. - write_submit() now checks SubmitConfig.n_chunks against the run directory's own RunMeta.n_chunks (fixed at prep time, and what rows_per_chunk is sized against) and raises a clear error on mismatch, instead of an uncaught IndexError deep in _job_walltimes. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:48:43 +02:00
lars	b8a4dc7d63	analyze: thread full model/training/rollout/dataset params to plots ... giant rollout now writes the checkpoint's complete model_config (incl. the router sub-dict), the sibling config.toml's [train]/[meta] sections, and every rollout CLI knob (weights, batch_size, escape_threshold, n_events, device, seed) into the YAML sidecar instead of a hand-picked subset. All of it flows through run_meta.json into each plot's own metadata.yaml for later comparison, while the figure subtitle itself shows a curated slice (hidden_dim, n_blocks, mode, conditioning, router, epoch, best_val_loss, steps/noise_dim) via new_figure's params option. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-27 13:35:02 +02:00
lars	61410ddee3	analyze: default run directory to <repo>/analysis_runs, gitignored ... giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.	2026-07-27 11:29:35 +02:00
lars	a88b21ef70	style: ruff format	2026-07-27 11:19:06 +02:00
lars	4d6101dcd7	analyze: recalibrate condor walltime model from real cluster timings ... The prior _COST_MODEL/_FIXED_OVERHEAD_S were fit only against local synthetic benchmarks (up to 2M rows/side), which can't see docker pull or real /ceph read latency and wildly overestimated real jobs (~1200-1800s predicted vs 50-320s median actual, from condor_history on production run 563f5ee3, --chunks 4, ~254M total rows). Refit each spec's per-row rate through the origin against its median real wall-clock time (not max, to avoid baking a few /ceph-contention spikes into a rate that would then wrongly scale with dataset size), and raised RUNTIME_SAFETY_MARGIN to compensate for that same contention risk instead.	2026-07-27 11:06:40 +02:00
lars	c7194701f6	analyze: raise default condor job memory request to 8192 MB ... 4096 MB was too tight: a resubmitted run held ~31 jobs spread evenly across nearly every plot family and chunk index with "Docker job has gone over memory limit of 4224 Mb", not one specific spec, so the generic per-chunk data footprint needed more headroom.	2026-07-27 10:31:46 +02:00
lars	dbd5c7e083	analyze: default condor docker image to alma9-gridjob ... mschnepf/slc7-condocker's ancient glibc/libstdc++ can't load current numpy/polars wheels from a uv-synced .venv (ImportError: CXXABI_1.3.9 not found). Switch the default to cverstege/alma9-gridjob, a modern EL9-based image.	2026-07-27 10:14:15 +02:00
lars	fa59443339	analyze: run condor compute jobs via .venv/bin/giant, not uv run ... uv isn't installed on the HTCondor worker docker image, so `uv run` fails there. giant is already an installed console script in the repo's uv-synced .venv, so exec it directly instead. write_submit now fails fast with a clear message if .venv/bin/giant is missing.	2026-07-27 10:03:02 +02:00
lars	e380400fe9	analyze: estimate per-job HTCondor walltime from chunk row count ... Each condor job's +RequestWalltime used to be one flat 3600s default for every (plot, chunk), regardless of how much data it actually streams over. `prep` now records each chunk's rollout+reference row count, and `giant/analysis/runtime_estimate.py` turns that into a per-job estimate: a per-spec (intercept, seconds/row) cost model fit by `scripts/profile_analysis_costs.py` against synthetic mock data on this machine, plus a fixed overhead placeholder (docker/uv/shared-fs startup — unmeasurable here, no /ceph access) and a single RUNTIME_SAFETY_MARGIN multiplier. jobs.txt gains a walltime column and the submit description references it via $(walltime) instead of a constant.	2026-07-27 09:47:44 +02:00
lars	86fc46b5a8	analyze: chunk per-plot aggregation across HTCondor jobs ... Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.	2026-07-27 09:24:19 +02:00
lars	70d0f04326	analyze: normalize pdg dtype in open_side to fix rollout/reference concat ... Rollout output and the reference file's ROOT-derived parquet disagree on pdg's integer width (Int32 vs Int64), which only surfaced downstream as a pl.concat SchemaError in build_context's pdg-count merge. Cast to a canonical Int64 at the single scan entry point instead.	2026-07-24 14:32:06 +02:00
lars	60c2ca1985	analyze: add MoE router gating/share diagnostic plots ... New "model" family in the gallery: router_gating (mean soft gate weight vs. pre-step energy, showing the router's soft decision boundaries) and router_share_by_pdg/router_share_by_process (stacked top-1 dispatch share by species / true physics process). Needs a live checkpoint's Router, so it's a documented exception to the rest of the package's polars/numpy-only contract; gracefully degrades to a placeholder for non-MoE checkpoints. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 13:32:25 +02:00
lars	77b2e9e5f4	analyze: drop stale ty:ignore on plotstyle import ... plotstyle is now a real installed package (from the git.larsbogner.de index) with its own type info, so the unresolved-import suppression is no longer needed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 11:17:41 +02:00
lars	1e7d7d7efd	analyze: show model/training params on rendered figures ... `giant rollout` now records the checkpoint's architecture (mode, hidden_dim, n_blocks, emb_dim, dropout, conditioning) plus training_epoch and best_val_loss in its YAML sidecar, using data already loaded from the checkpoint. condor.py carries those through run_meta, and render.py passes them to plotstyle's new_figure(params=...) so every plot's subtitle shows what produced it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 09:54:21 +02:00
lars	8ff70e3c87	analyze: drive prep/submit from the rollout YAML sidecar ... `giant analyze prep` / `submit` now take the `giant rollout` YAML sidecar as their only positional input instead of explicit --rollout/--reference/--out-dir. The YAML's `output`/`dataset` keys name the rollout parquet and its seed file (the reference truth), and the rest of the sidecar (checkpoint, geometry oracle, cutoffs) flows into every plot's gallery metadata. prep derives its own run directory next to the rollout parquet (<...>/analysis_<id>/) holding shared.json, run_meta.json, reduced/, plots/. compute-one and render now take just --run-dir / a run-dir argument and read the resolved paths + metadata from run_meta.json, so the condor wrapper no longer threads file paths. open_side scans a directory of reference shards via glob. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:56:36 +02:00
lars	f4c2545e8b	Rewrite analysis as streaming rollout-vs-reference plotting pipeline ... Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth diagnostics) with a lean giant/analysis/ package that compares one autoregressive `giant rollout` for a checkpoint against a held-out miniCaloSim reference file, and generates publication-styled plots in parallel on HTCondor. Rollout output and a raw reference file share a world-frame physical column subset under identical names, so the old ALR/local-frame decode machinery is gone — everything is world-frame mm/MeV. - sources.py: canonical LazyFrames, synthetic-termination-row filtering, the secondary view (rollout generation>0 tracks vs reference sec_*_list). - reduce.py: streaming primitives — a single hist1d group_by pass, per-event scalars, edep-weighted depth/transverse profiles, species share, leakage. - context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material group sets once into shared.json, so each compute job is one pass, no range scan (histogram efficiency). - catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg, material}, per-event totals, shower profiles, species/leakage, secondaries. - render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata. - condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit): one job per plot, compute/render split (workers polars-only, no LaTeX). Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover the reduce primitives, catalog id uniqueness + compute, condor submit, and a guarded render smoke test. Delete the two predict-diagnostics notebooks. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-23 17:38:09 +02:00