From 1e92902c8dc6f60a06e0f46b9104f454563c3445 Mon Sep 17 00:00:00 2001 From: Lars Bogner Date: Mon, 24 Aug 2026 15:30:14 +0200 Subject: [PATCH] Backfill CHANGELOG.md for v0.2.0-v0.3.2 The automated changelog (gitea #50) deliberately started fresh with no backfill; this reverses that call now that it's wanted. v0.2.0-v0.3.2 are generated from tag history via git-cliff/cliff.toml, matching the format of existing entries. v0.3.3 was bumped but never tagged, so its commits stay folded into the existing v0.3.4 entry. The v0.2.0 range (198 uncurated pre-automation commits) is hand-curated to drop duplicate commits and dev-log noise (WIP markers, incomplete-validation runs, repeated "Apply ruff format"). --- CHANGELOG.md | 477 ++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 476 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e5be149..63b31a0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -65,4 +65,479 @@ - Document CI_TOKEN's write:repository scope requirement [gitea #50](https://git.larsbogner.de/lars/giant/issues/50) -# Changelog +## [0.3.2] - 2026-08-17 + +### Added + +- Add configs/baseline.toml as the kept reference model + + +### Fixed + +- Clamp analysis histogram bins before the i32 cast, not after [gitea #61](https://git.larsbogner.de/lars/giant/issues/61) + +- Clip raw predicted log_mass in decode_secondaries [gitea #54](https://git.larsbogner.de/lars/giant/issues/54) + + +### Changed + +- Let dwarf warm-cache take --config so it can't under-warm a config's cache keys [gitea #59](https://git.larsbogner.de/lars/giant/issues/59) + +- Implement n_sec.mode = "stop_token" for the AR secondary decoder [gitea #40](https://git.larsbogner.de/lars/giant/issues/40) + +- Bump patch version to 0.3.2 + +## [0.3.1] - 2026-08-14 + +### Added + +- Add an Objective registry for the flow/ddpm/wgan generator choice [gitea #32](https://git.larsbogner.de/lars/giant/issues/32) + + +### Changed + +- Make trunk architecture selectable via a registry [gitea #33](https://git.larsbogner.de/lars/giant/issues/33) + +- Make ResBlock's conditioning-injection mechanism selectable [gitea #34](https://git.larsbogner.de/lars/giant/issues/34) + +- Make HistoryEncoder a pluggable registry, like Router/Objective [gitea #35](https://git.larsbogner.de/lars/giant/issues/35) + +- Deduplicate n_sec_head/type_head MLPs into build_mlp_head [gitea #36](https://git.larsbogner.de/lars/giant/issues/36) + +- Give the cond_cat/cond_cont column layout one owner [gitea #37](https://git.larsbogner.de/lars/giant/issues/37) + +- Give Stage1Model/Stage2OneShot/Stage2Autoregressive a shared StageModel base [gitea #39](https://git.larsbogner.de/lars/giant/issues/39) + +- Pass ConditioningAxisConfig/ParticleTypeConfig themselves instead of raw dicts [gitea #38](https://git.larsbogner.de/lars/giant/issues/38) + +- Bump patch version to 0.3.1 + +## [0.3.0] - 2026-08-13 + +### Added + +- Add v0.3.0 design doc: Stage-2 autoregressive redesign + +- Add pytest-cov to dev deps and run coverage in CI + +- Add coverage for router-center seeding, geometry batch reader, material topN cache, and setup-cache corruption paths + +- Add render.py coverage: figure params, router diagnostics plots, gallery/condor glue + +- Add unknown-key validation to config.toml merge (issues.md Issue 2) + +- Add consumed-keys audit test (issues.md Issue 5) + + +### Fixed + +- Fix test_render_all_run_gallery_invokes_subprocess clobbering LaTeX's own subprocess.run + + +### Removed + +- Remove issues.md + + +### Changed + +- Refine v0.3.0 design: defaults, deferred scope, open questions + +- Document the differentiability position and its validation obligation + +- V0.3.0 step 1: new nested config schema, v0.2 migration shim + +- V0.3.0 step 2: network.py refactor to composable stage models + +- V0.3.0 step 3: per-stage train.py trainers + pipeline.py/cli.py rewrite + +- V0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" + +- V0.3.0 step 5: Stage2Autoregressive (history=markov) + ยง11.4 grad instrumentation + +- V0.3.0 step 6: sample.py/rollout.py AR generation + class->PDG decode + +- V0.3.0 step 7: AttentionHistory (KV-cached) + scheduled/never teacher forcing + +- V0.3.0 post-implementation audit: resolve all 9 tracked discrepancies + +- Refactor train.py into giant/training/ around a metrics collector + +- Silence the fork-safety warning from num_workers>0 pipeline tests + +- Deduplicate giant/training/trainers.py shared per-stage logic + +- Rewrite README for v0.3.0 architecture, quick start, and data columns + +- Bump version to 0.3.0 + +- Delete docs/v0.3.0-design.md and strip all references to it + +- Apply ruff format + +- Downgrade coverage-report upload to actions/upload-artifact@v3 + +- Bump ruff line-length to 120 and reformat + +- Make config dataclasses the single source of truth for DEFAULT_CONFIG + +- Extract giant train/new-run's CLI override mapping into a table-driven function (issues.md Issues 3 & 4) + +- Mark issues.md Issues 3 & 4 as fixed + +- Extract predict/rollout's duplicated inference bootstrap into giant.checkpoint_io (issues.md Issue 5) + +- Mark issues.md Issue 5 as fixed + +- Unify the two v0.2->v0.3 migration surfaces (issues.md Issue 6) + +- Type the data/model/training batch contracts with NamedTuples (issues.md Issue 7) + +- Split giant/model/network.py into giant/model/ (issues.md Issue 8) + +- Move scripts/ to giant/tools/ (issues.md Issue 9) + +- Reject stage2_model.stage1_context = 'sampled' as unimplemented (issues.md Issue 1) + +- Honour wgan.critic_hidden_dim/critic_n_res_blocks in build_critics [gitea #28](https://git.larsbogner.de/lars/giant/issues/28) + +- Validate stage2_model.autoregressive.order in validate_config [gitea #30](https://git.larsbogner.de/lars/giant/issues/30) + +- Decouple secondary-species vocabulary from conditioning.particle.emb_dim [gitea #29](https://git.larsbogner.de/lars/giant/issues/29) + +- Skip router auxiliary loss compute when their lambda is 0 [gitea #31](https://git.larsbogner.de/lars/giant/issues/31) + +## [0.2.0] - 2026-08-04 + +### Added + +- Add CLAUDE.md with architecture overview and dev commands + +- Add streaming data pipeline and giant CLI entry point + +- Add giant predict command + +- Add ROOT-to-parquet conversion script with convert dependency group + +- Add post_pos as a model target via travel_dir decomposition + +- Add --coord local mode to predict for raw-space prediction debugging + +- Add KL divergence to marginal validation and hook it into the training loop + +- Add graceful shutdown on SIGINT/SIGTERM + +- Add configurable dropout to ResBlocks + +- Add giant.analysis module for notebook-based model quality diagnostics + +- Add lazy polars I/O and duplicate KL/constraint checks for giant.analysis + +- Add ruff and ty as dev dependencies, fix lint/type findings + +- Add linear warmup before cosine LR decay + +- Add --batch-size auto to estimate batch size from free GPU memory + +- Add hyperparameter scan + +- Add --batch-size auto to predict, matching train + +- Add tqdm progress bar to predict + +- Add KL bar plots and sample_frac to load_predicted_local; ignore root parquet scratch files + +- Add event-level shower observables to giant.analysis + +- Add total length traveled per event to event observables + +- Add pdg energy/length contribution pie plots + +- Add export script for Tier 4 event-level/pdg-share plots + +- Add mean/median deposited energy and step length plots per event + +- Add export script for ETP group-update presentation plots + +- Add photon edep export scripts and per-step presentation plots + +- Add tooling for a versioned geant_steps dataset layout + +- Add --copy mode to migrate_geant_steps.py + +- Add update-manifest and create-manifest subcommands to bump_dataset_version + +- Add --to flag for bump-gen/bump-schema and --gen flag for update-manifest + +- Add disk usage summary to dwarf status + +- Add file counts and reference tracking to dwarf status + +- Add --comment option to predict, recorded in YAML sidecar + +- Add energy-conservation PoC ODE-step comparison scripts + +- Add autoregressive shower rollout driver + +- Add fast slab lookup for the GeometryOracle, replacing knn as the default + +- Add load_rollout_vs_truth to compare rollouts against held-out truth data + +- Add mixture-of-experts routing prototype for Stage 1 and Stage 2 + +- Add ProcessRouter for physics-process-based expert gating + +- Add PdgRouter for particle-type-based expert gating + +- Add ComposedRouter for multi-axis MoE gating + +- Add EMA weights, weight decay, step-based LR schedule, and grad-norm logging to training + +- Add WGAN-GP mode as a throwaway fast-eval experiment + +- Add router gating diagnostic for MoE checkpoints + +- Add Gitea Actions CI pipeline + +- Add configs for router energy (embedding/physical) and WGAN baseline runs + +- Add opt-in Weights & Biases logging for the training loop + +- Add test coverage for resolve_expert_dims + +- Add regression coverage for vocab/process index-map builders + +- Add dwarf warm-cache to precompute the setup-stage sidecar + +- Add giant new-run to scaffold a config.toml + run dir ahead of training + +- Add learnable per-expert width and shared temperature to EnergyRouter + +- Add opt-in straight-through Gumbel-softmax combine weights to MoE router + +- Add gumbel router configs sweeping learnable-knob combinations + +- Add gumbel/learn_centers/learn_width/learn_temperature to out-dir naming + +- Add bigger WGAN config (hidden_dim=512, n_blocks=6) + +- Add data-integrity guards against silent NaN/Inf propagation and races + + +### Fixed + +- Fix column names to match actual parquet schema + +- Fix installed torch version to be compatible with cuda drivers + +- Fix miniCaloSim link in README + +- Fix giant.analysis import after Phase 2 dataset API changes + +- Fix silent failure modes surfaced by extensive code review + +- Fix ruff, ty, and pytest failures; apply ruff format + +- Clamp n_sec classification label to K_MAX + +- Fix rollout edep mismatch and add truth overlay to Tier 4 observables + +- Fix crashes in physical-property conditioning edge cases + +- Fix router experts silently ignoring --hidden-dim/--n-blocks + +- Fix conditioning="physical" so it can actually generalize past training vocab + +- Fix training-loop checkpoint/resume and WGAN bugs + +- Fix stale-partial reuse and n_chunks mismatch in analysis condor pipeline + +- Fix CLI/tooling robustness gaps and dedupe the Conditioning enum + +- Fix test_write_submit_requires_synced_venv for active-venv resolution + + +### Removed + +- Remove scripts/train.py in favor of the giant train CLI + +- Drop orphaned child tracks instead of nulling secondary targets + + +### Changed + +- Initial commit: giant surrogate model with two-phase roadmap in README + +- Implement Phase 1: full data pipeline, model, training, and config support + +- Handle material column as string type + +- Rename pre_energy/post_energy columns to pre_E/post_E + +- Rename direction columns from pre_dir_x/y/z to pre_dx/dy/dz + +- Batch StreamingStepsDataset internally instead of per-row collate + +- Dedup training pipeline, add seeding/resume and per-epoch metrics logging + +- Split torch into cpu/cuda extras and pin dependency version bounds + +- Apply ruff format and document lint/type tooling in CLAUDE.md + +- Update README to match current architecture and tooling + +- Make sampler step count configurable for validation + +- Calibrate auto batch size separately for inference vs training + +- Skip rows with unknown PDG codes during predict + +- Buffer predict rows across row-group boundaries before inference + +- Export plots for knowledge base + +- Rework validation notebook with markdown sections and Tier 4 plots + +- Allow steps_to_parquet.py to accept multiple ROOT input files + +- Encode edep/secondary/post energy as a conservation-constrained simplex + +- Expose dataset/conversion scripts as uv entry points + +- Restrict holdout overlap check to holdout vs dev/full only + +- Route predict output to UUID-named parquet with YAML reference sidecar + +- Implement Phase 2: secondary particle prediction + +- Unify dataset/tooling scripts into a single `dwarf` Typer CLI + +- Fold --to/--gen dataset-versioning flags into the dwarf CLI + +- Prefix default train output dir with current date + +- Color-code dwarf status output by tree level + +- Show VERSIONS.md reason extracts in dwarf status + +- Wire up predict CLI to load and run the Stage-2 sec_decoder + +- Wire up n_sec/species/energy-fraction validation for Stage 2 + +- Detach Stage-2 type-embedding target to stop self-referential collapse + +- Weight Stage-2 secondary loss equally between direction and type-embedding dims + +- Recalibrate batch-size estimate for the post-Phase-2 model size + +- Update CLAUDE.md and README for the implemented Phase 2 model + +- Error on missing secondary lists instead of silently zeroing Stage-2 targets + +- Derive a unique per-job seed for minicalosim shard generation + +- Rescale secondary energies to exactly consume the e_sec budget + +- Support --energy-gev in dwarf make-root for the new minicalosim energy arg + +- Stream giant rollout output instead of buffering the whole run + +- Scale auto batch-size estimate by MoE expert count during training + +- Rewrite analysis module as a lean, fully-streaming pipeline + +- Reimplement rollout-vs-truth comparison on the streaming analysis module + +- Condition on material/particle physical properties instead of learned embeddings + +- Ignore the scratchpad working directory + +- Quote the on: key in the CI workflow + +- Split CI lint stage into parallel jobs + +- Rewrite analysis as streaming rollout-vs-reference plotting pipeline + +- Analyze: drive prep/submit from the rollout YAML sidecar + +- Analyze: show model/training params on rendered figures + +- Deps: install plotstyle from git.larsbogner.de package index + +- Analyze: drop stale ty:ignore on plotstyle import + +- Test: replace prep(**_CTX) splat with a typed _prep helper + +- Analyze: add MoE router gating/share diagnostic plots + +- Chore: remove stray CUDA sanity script and stale Phase 2 planning doc + +- Docs: document compute environment, WGAN/MoE status, and condor-gpu-train-rollout + +- Analyze: normalize pdg dtype in open_side to fix rollout/reference concat + +- Analyze: chunk per-plot aggregation across HTCondor jobs + +- Analyze: expose bin/pdg options on `analyze submit` + +- Analyze: estimate per-job HTCondor walltime from chunk row count + +- Analyze: run condor compute jobs via .venv/bin/giant, not uv run + +- Analyze: default condor docker image to alma9-gridjob + +- Analyze: raise default condor job memory request to 8192 MB + +- Analyze: recalibrate condor walltime model from real cluster timings + +- Transforms: pad legacy cond normalizers for pre-physical-conditioning checkpoints + +- Analyze: default run directory to /analysis_runs, gitignored + +- Docs: record first MoE router rollout benchmark result in the roadmap + +- Router: seed EnergyRouter centers from data quantiles instead of a fixed linspace + +- Docs: note the EnergyRouter centers_init fix in the roadmap + +- Analyze: thread full model/training/rollout/dataset params to plots + +- Ci: share one uv sync across jobs, gate tests on lint+type-check, sync tag/version on release tags + +- Ci: replace unsupported artifact sharing with a bind-mounted uv cache + +- Ci: stop setup-uv from overriding UV_CACHE_DIR + +- Ci: re-pin UV_CACHE_DIR after setup-uv, which exports its own value regardless of enable-cache + +- Ci: set UV_LINK_MODE=copy to silence the cross-filesystem hardlink warning + +- Log batch-level metrics to W&B, not just per-epoch summaries + +- Log router health, WGAN grad-norm split, n_sec accuracy, GPU/throughput to W&B + +- Persist global_step across --resume so W&B step stays monotonic + +- Timestamp default checkpoint dir to avoid W&B run-id collisions + +- Skip empty-slice mean/std in sec phys validation print + +- Speed up giant train's setup stage + +- Speed up _WelfordAccumulator's per-chunk update + +- Make default checkpoint out_dir name reflect only non-default hyperparams + +- Cache giant train's setup stage in a sidecar file + +- Pass --seed through to the train/val event split + +- Offset event_id per file to avoid cross-file collisions + +- Store a quantile grid instead of a raw reservoir sample in the setup cache + +- Scope wandb run config to only-active hyperparameters + +- Resolve giant condor wrapper from the active venv, not a hardcoded path + +- Bump version to 0.2.0