304 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
lars	417b741484	Merge pull request 'Add giant analyze metrics plots for training progress (gitea #75)' (#78) from fix/issue-75 into master ... Reviewed-on: #78	2026-08-24 11:32:34 +02:00
lars	37d73e6578	Merge branch 'master' into fix/issue-75	2026-08-24 11:32:19 +02:00
gitea-actions	ff204732d7	chore: update changelog for v0.3.7 [skip ci] v0.3.7	2026-08-24 09:31:26 +00:00
gitea-actions	02ed4e531c	chore: bump version 0.3.6 -> 0.3.7 [skip ci]	2026-08-24 09:31:25 +00:00
lars	1b6c8b33b7	Merge pull request 'Add rollout-quality distance, confusion, containment and router plots (gitea #76)' (#79) from fix/issue-76 into master ... Reviewed-on: #79	2026-08-24 11:22:11 +02:00
lars	7560e2bff0	Fix LaTeX-unavailable skip check in analyze metrics smoke test ... CliRunner stores an uncaught exception in result.exception, not result.output, so the skip condition never matched and the test failed outright on CI machines without LaTeX installed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-24 11:15:51 +02:00
lars	ffb7c0cc2a	Add rollout-quality distance, confusion, containment and router plots (gitea #76) ... Picks 4 of the 7 catalog additions the issue proposed (the smaller-lift ones; 2D joint plots, PIT calibration, and the throughput/accuracy scatter are left for follow-up issues): - marginal_distance_summary: a var x grouping-axis KS-statistic heatmap, reusing the existing marginal hist1d compute and just adding a finalize — a single at-a-glance regression scorecard instead of N overlay plots. - n_sec_confusion: predicted (rollout) vs true (reference) secondary count per event, paired by event_id since a rollout is seeded from the same events as its reference file. Needed a new zero-filling primitive (reduce.sec_count_by_event) since a plain group_by over secondary rows silently drops zero-secondary events. - shower_containment_depth_{90,95}: per-event depth containing 90%/95% of deposited energy, derived from the same per-event depth-bin matrix the longitudinal profile already computes. - router_specialization: max gate weight vs energy per side, summarizing router_gating's full stacked area into the one trend line the roadmap's MoE writeup describes (the ~60-65% ceiling), to make a future lambda_balance>0 retrain's effect on specialization checkable at a glance. Both new heatmap-shaped plots (distance summary, confusion matrix) share one new "heatmap" Reduced kind/renderer rather than two near-identical ones. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-24 11:12:16 +02:00
lars	bdebd83c8b	Add giant analyze metrics plots for training progress (gitea #75) ... MetricsCollector writes one row per epoch to <run_dir>/metrics.csv, but nothing read or plotted it. giant/training/plots.py reads the CSV header dynamically (the column set varies by run: flow/ddpm vs wgan, routed vs not) and renders loss/lr/accuracy/grad-norm/router/wgan-balance/throughput plots with the same plotstyle conventions giant/analysis/render.py uses, skipping any figure whose columns aren't present for a given run. Wired up as `giant analyze metrics <run_dir>`, writing PDFs into the same gitignored analysis_runs/ directory `analyze prep`/`submit` already use (derive_metrics_dir mirrors derive_run_dir) rather than into the training run directory itself.	2026-08-24 10:55:15 +02:00
gitea-actions	97f5bbf9f0	chore: update changelog for v0.3.6 [skip ci] v0.3.6	2026-08-24 08:02:49 +00:00
gitea-actions	060353ea4a	chore: bump version 0.3.5 -> 0.3.6 [skip ci]	2026-08-24 08:02:48 +00:00
lars	b3f28e98af	Merge pull request 'Give CriticModel a registry-built trunk and StageModel base (gitea #57)' (#74) from fix/issue-57 into master ... Reviewed-on: #74	2026-08-24 09:57:58 +02:00
lars	4b2e0ba98e	Give CriticModel a registry-built trunk and StageModel base (gitea #57) ... CriticModel was the one stage-shaped class left out of the trunk-registry (gitea #33), block-conditioning-registry (gitea #34), and StageModel-base (gitea #39) refactors: it hand-rolled a plain ResBlock stack, so a routed/FiLM/AdaLN trunk was available to every generative stage model except the critic competing against them under WGAN-GP. CriticModel now subclasses StageModel (reusing its cond_enc construction, and a stage-2 context-fusion helper factored out of Stage2OneShot onto the base) and builds its body via build_trunk (output width 1) instead of a bespoke ResBlock loop, so trunk.type/trunk.block_conditioning now affect the critic too. Each stage's critic inherits its own generator's trunk config rather than a new critic_trunk config key, mirroring the existing critic_hidden_dim/critic_n_res_blocks "0 = inherit from generator" pattern. Router mixing (MoE) for the critic stays out of scope. Since CriticModel is training-only and never persisted for inference, and WGAN-GP is still unbenchmarked, its state_dict shape has no back-compat burden. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-24 09:48:26 +02:00
gitea-actions	1675052ecd	chore: update changelog for v0.3.5 [skip ci] v0.3.5	2026-08-24 07:37:23 +00:00
gitea-actions	eb9d331bea	chore: bump version 0.3.4 -> 0.3.5 [skip ci]	2026-08-24 07:37:22 +00:00
lars	12689cf5b6	Merge pull request 'Add "none" variants for router, history, and trunk (gitea #45)' (#73) from fix/issue-45 into master ... Reviewed-on: #73	2026-08-24 09:32:30 +02:00
lars	732d5f1cd2	Add "none" variants for router, history, and trunk (gitea #45) ... Turns "is this component earning its parameters?" into a one-line config flip for each of the three pluggable network components: - router.type = "none" (NoneRouter, giant/model/routers.py): still builds n_experts expert trunks via RoutedTrunk, but replaces the learned gate with a uniform 1/n_experts weight for every row — no centers/embeddings/classifier. Distinct from router.enabled=false (which drops routing/mixing entirely): this isolates whether the *learned routing signal* specifically is earning its parameters, holding expert count fixed. - stage2_model.autoregressive.history = "none" (NoHistory, giant/model/history.py): ignores feat/has_prev entirely and always returns zeros, ablating whether the AR decoder's history conditioning earns its parameters. Already validated for free by gitea #35's generic HISTORY_REGISTRY membership check. - trunk.type = "linear" (LinearTrunk, giant/model/trunks.py): a bare nn.Linear(in_dim + cond_dim, out_dim) body, no ResBlock stack. Per gitea #33's design, this composes for free with router.enabled=true ("mixture of trivial linear experts"). Both blocking issues (#33 trunk registry, #35 pluggable history encoder) are closed, so this was unblocked.	2026-08-24 09:22:36 +02:00
gitea-actions	ef8a2f4e55	chore: update changelog for v0.3.4 [skip ci] v0.3.4	2026-08-23 19:50:10 +00:00
gitea-actions	d61a9b7661	chore: bump version 0.3.3 -> 0.3.4 [skip ci]	2026-08-23 19:50:08 +00:00
lars	dc16265e18	Merge pull request 'Document CI_TOKEN's write:repository scope requirement (gitea #50)' (#72) from fix/issue-50 into master ... Reviewed-on: #72	2026-08-23 21:39:52 +02:00
lars	aff0ef881f	Document CI_TOKEN's write:repository scope requirement (gitea #50) ... The prior e2e run (task 1348) failed on the bump-version job's push step with a 403 Forbidden — CI_TOKEN lacked write access. Note this on the checkout step so the requirement isn't lost, now that the token has been rescoped. Trivial commit to re-open a merge request and re-run the job end to end. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-23 21:38:41 +02:00
lars	d0cbcbce80	Merge pull request 'Auto-bump patch version, tag, and update changelog on merge to master (gitea #50)' (#71) from fix/issue-50 into master ... Reviewed-on: #71	2026-08-18 10:50:16 +02:00
lars	10a57322f9	Merge branch 'master' into fix/issue-50	2026-08-18 10:42:32 +02:00
lars	c09ebd2410	Merge pull request 'Add class-balanced secondary particle-type loss (gitea #44)' (#70) from fix/issue-44 into master ... Reviewed-on: #70	2026-08-18 10:41:42 +02:00
lars	5b478d2831	Auto-bump patch version, tag, and update changelog on merge to master (gitea #50) ... Version bumps and release tags were entirely manual; the only CI automation was sync-version-on-tag, which corrects pyproject.toml if a hand-pushed tag drifted. This flips that: a new bump-version job (needs the four existing checks, gated to actual merge commits on master via HEAD^@'s parent count so direct/squash/rebase pushes are untouched) uses bump-my-version to auto-bump the patch version when a merged branch didn't already bump it itself, then generates a changelog entry with git-cliff and pushes a matching vX.Y.Z tag. git-cliff's cliff.toml is tuned to this repo's plain imperative commit style (no feat:/fix: prefixes): commits are grouped Added/Fixed/Removed/Changed by leading verb, "(gitea #N)" is linkified, and merge/[skip ci] commits are dropped. Per user decision during planning: the changelog generator folds in @lars's comment on the issue (asking to fold in changelog generation rather than deferring it), and CHANGELOG.md starts fresh with no backfill of v0.2.0-v0.3.3. sync-version-on-tag is left untouched as the safety net for hand-tagging. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-18 10:40:19 +02:00
lars	de805fb0a7	Merge pull request 'Offset event_id across multi-shard reference reads in giant analyze (gitea #22)' (#69) from fix/issue-22 into master ... Reviewed-on: #69	2026-08-18 10:23:20 +02:00
lars	fce47b128c	Add class-balanced secondary particle-type loss (gitea #44) ... The v0.3.0 pivot exists because the 2026-08-03 WGAN rollout benchmark produced zero photon secondaries and ~4M hallucinated antineutrinos — even with a correctly-sized top-N species vocabulary (gitea #29), plain cross-entropy over a class distribution spanning orders of magnitude still under-predicts rare-but-physical species. stage2_model.particle_type.class_weighting = "none" | "inverse_freq" (default "none", fully back-compat) weights the stage-2 type head's CE loss (FlowDDPMStageTrainer._type_loss) by inverse class frequency, normalized to mean 1 so switching it on doesn't rescale the type loss against particle_type.lambda / the generator loss it's summed with. The per-class counts the weighting needs don't already exist despite the issue's premise: _topn_plus_other_map (giant/data/loader.py) previously kept counts only for keys folded into "other", dropping the kept classes' counts on the floor. TopNMap now carries class_counts (index -> count), round-tripped through the setup-cache sidecar (format version bumped 3->4, since existing sidecars have none) and through checkpoints (tolerantly — a pre-#44 checkpoint decodes to {}, since only training-time loss weighting reads it, not inference). Decisions made during planning (with the user): dropped "effective_num" from the issue's proposed three-way enum (no beta hyperparameter to design around) — final domain is "none" | "inverse_freq". Weights are mean-1-normalized. validate_config rejects class_weighting != "none" combined with particle_type.target != "onehot" or stage2_model.generator == "wgan" (both have no class CE to weight), following the #28/#30 dead-key-must-not-go-silent convention. Branch fix/issue-44 off master. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-17 23:02:50 +02:00
lars	c1e6ffd8c6	Offset event_id across multi-shard reference reads in giant analyze (gitea #22) ... giant/analysis/sources.py's open_side scanned a reference directory of parquet shards with a bare glob and never offset event_id across them. Each shard is a separate Geant4 job whose own event_id numbering restarts from 0, so events from different shards collided on the same event_id, corrupting every downstream per-event grouping and the event_id % n_chunks condor chunking — the same root cause already fixed on the training/rollout side via giant/data/loader.py's per-file event_id_offset. open_side's reference branch now uses find_parquet_files (the same deterministically ordered file lister giant rollout's _seed_from_data uses) and offsets each shard's event_id via a join on polars' include_file_paths, so both sides of a comparison agree on what an event_id means. Two incidental behaviour changes come along for free: .manifest references now work (they crashed before), and the directory glob narrows from recursive **/*.parquet to top-level *.parquet, matching the file list rollout itself used to assign offsets — a deliberate choice, since a differing file list would make the two sides' offsets disagree again in a subtler way. No overflow guard on the per-shard offset stride (unlike loader's _offset_event_id): checking it here would cost an eager event_id-column read per shard in every condor compute job, and giant rollout already runs that check over the same file list when producing the seed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-17 15:53:02 +02:00
lars	f60af64d00	Merge pull request 'Add bf16 autocast to the training loop (gitea #47)' (#68) from fix/issue-47 into master ... Reviewed-on: #68	2026-08-17 15:45:51 +02:00
lars	78978769f6	Add bf16 autocast to the training loop (gitea #47) ... giant/ had no autocast/GradScaler/torch.compile anywhere despite the project's ~10x-native-Geant4 eval-budget target. This adds bf16 mixed precision to the training step (both FlowDDPMStageTrainer and WGANStageTrainer) via a new train.precision config key ("fp32" default, "bf16" opt-in) and giant.training.amp.resolve_autocast. torch.compile is a separate, much larger surface (data-dependent routed dispatch, the autoregressive sampler's per-token control flow, arbitrary rollout batch sizes) and is left for a follow-up issue, per discussion. Scope decisions made during planning: - fp32 + bf16 only, no fp16/GradScaler. fp16 breaks two things in this codebase: routers.py's three 1e-8 epsilons sit below fp16's ~6e-8 subnormal floor, and gradient_penalty's grad norm overflows fp16's range at ordinary early-WGAN-GP gradient magnitudes. Every training GPU in the fleet (A100/L40S/H200/RTX 4070) has native bf16; only pre-Ampere V100s would need fp16. - resolve_autocast raises loudly if bf16 is requested on hardware that can't do it, rather than silently falling back to fp32. - Autocast wraps the training step only; val_loss (and the best-checkpoint selection it drives) stays fp32 so it's comparable across every run recorded so far. - _route_forward's mixture accumulator (giant/model/trunks.py) was a hard-fp32 torch.zeros with no dtype, so under autocast a RoutedTrunk silently returned a different output dtype than an unrouted ExpertTrunk purely because router.enabled was set. Fixed to match the experts' own dtype; the gate weights (forced fp32 for their own numerical stability) are cast down before combining, so the mixture's numerics stay solid without reintroducing the dtype split. - Added explicit fp32 guards (autocast(enabled=False)) around spots that are correct in fp32 but degrade quietly rather than crash in bf16: the router's balance/entropy losses and gate softmax, the stage-2 stick-breaking cumprod, and gradient_penalty's double-backward + grad norm. Benchmarked on the local RTX 4070 against configs/baseline.toml's hyperparams (hidden_dim 512/6 blocks, bs 4096) on a synthetic dataset: bf16 gave 1.05-1.35x training throughput and 18-33% lower peak GPU memory across one-shot/routed/autoregressive stage-2 configs, with the autoregressive path (the dominant cost per baseline.toml) benefiting most on both axes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 15:36:12 +02:00
lars	692acd77eb	Merge pull request 'Add per-stage init_from/freeze (gitea #42)' (#67) from fix/issue-42 into master ... Reviewed-on: #67	2026-08-17 14:48:38 +02:00
lars	e8842c56d7	Bump patch version to 0.3.3 ... Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 14:25:51 +02:00
lars	87e37ebe14	Add per-stage init_from/freeze (gitea #42) ... stage{1,2}_model.active = false already trains one stage alone, but the checkpoint it writes holds only that stage, so giant rollout refuses it -- the "retrain stage 2 alone against a fixed, known-good stage 1" experiment the 2026-08-03 species failure calls for wasn't runnable end to end. Adds stage{1,2}_model.init_from (a checkpoint .pt to load this stage's weights from before training) and .freeze (never update them), symmetric across both stages. Both stages stay active = true, so both get built and both land in the output checkpoint -- the frozen stage is merely initialized from disk instead of from scratch. Decisions made during planning: - Soft freeze: forward/backward still run every batch (loss/grad_norm stay meaningful, no autograd special-casing), only optimizer.step() (and, for the frozen stage, lr_sched.step()/EMA update) is skipped -- weights are byte-identical for the whole run. This is StageTrainer._step_optimizer, shared by the non-adversarial path and both halves (generator + critic) of the WGAN path, so a frozen WGAN stage's critic freezes too. - validate_config requires init_from whenever freeze = true, unless the run is a --resume (a resumed frozen stage's weights come from the resume checkpoint instead) -- freezing a randomly-initialized model is almost certainly a mistake. - CLI flags on both `giant train` and `giant new-run` (--stage{1,2}-init-from/--stage{1,2}-freeze), matching every other per-stage model knob's existing treatment. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 14:23:20 +02:00
lars	8290e350b8	Merge pull request 'Implement stage2_model.stage1_context = "sampled" (gitea #41)' (#66) from fix/issue-41 into master ... Reviewed-on: #66	2026-08-17 13:40:54 +02:00
lars	48faaee79d	Implement stage2_model.stage1_context = "sampled" (gitea #41) ... Stage 2 was trained on ground-truth stage-1 outcomes but deployed on sampled ones, and in a rollout that gap compounds over every step of every track — the same train/inference gap teacher_forcing="scheduled" already closes within stage 2, just never applied at the stage boundary. "sampled" was declared in the schema but rejected loudly by validate_config as unimplemented; this lands the real implementation. Mirrors the existing scheduled-sampling precedent rather than a hard switch: new stage2_model.ctx_p_start/ctx_p_end (defaults 1.0 -> 0.0) linearly ramp P(condition on ground truth) from epoch 0 to the final epoch, so stage 2 doesn't chase a wildly moving stage-1 target early in training. Per the plan discussed with the user: the sample is drawn from stage 1's sampling_model() (EMA weights when present, matching what inference actually deploys), mixed per example via a Bernoulli draw (never blended within a row), and validation always uses the ground truth regardless of the schedule. Fixes a latent bug the same pattern would otherwise have hit: every sampler in giant/sample.py flips its model to .eval() with no restore, so sampling from the raw (non-EMA) stage-1 model mid-step now explicitly restores its .training flag afterward to avoid silently corrupting stage 1's own training mode for the rest of the epoch. validate_config now enforces stage1_context in {"truth", "sampled"}, requires both stages active for "sampled" (nothing to sample from otherwise), range-checks ctx_p_start/ctx_p_end, and rejects the ctx_p_start = ctx_p_end = 1.0 configuration as an unadvertised no-op identical to "truth". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 12:27:13 +02:00
lars	09bea2cbff	Merge pull request 'Add giant model summary command (gitea #46)' (#65) from fix/issue-46 into master ... Reviewed-on: #65	2026-08-17 12:08:59 +02:00
lars	cc9646f279	Add giant model summary command (gitea #46) ... giant model summary --config config.toml builds the resolved Stage1/Stage2 graph from a config with no dataset attached (pdg_vocab/mat_vocab are supplied as placeholders via --pdg-vocab/--mat-vocab, since the real training vocab is dataset-derived) and prints per-module parameter counts, trunk in/out widths, which heads exist, and which conditioning/stage1_model/stage2_model config keys actually shaped the build. The consumed-keys half uses differential probing rather than static identifier matching: build once for a fingerprint (submodule presence, every parameter's/buffer's shape+dtype, every plain scalar attribute a module stores on itself), then perturb one leaf at a time, rebuild, and compare. A changed fingerprint (or a raise) means the key is consumed; no change means it's inert *under this particular config* -- e.g. any stage1_model.router.* key when router.enabled=false. A curated _NOT_BUILD_TIME table separates keys legitimately owned by the trainer/sampler/rollout (loss weights, WGAN-GP hyperparameters, teacher-forcing schedules) from genuinely-inert ones, verified against those call sites. A few config keys branch on equality against one specific string literal (n_sec.owner=="stage1", n_sec.mode=="stop_token", particle_type.target=="physical"); a single generic sentinel probe missed all three since the config's current value and the sentinel landed in the same branch, so those three leaves get their real alternative value tried too (_STRING_ALTERNATIVES). giant.config.leaf_paths is promoted out of tests/test_config_consumed_keys.py (previously a private test-local duplicate) so both audits -- the static per-identifier one and this new runtime per-config one -- walk the exact same DEFAULT_CONFIG tree. ExpertTrunk/RoutedTrunk now also expose in_dim (out_dim already existed), needed to report trunk widths generically. Decisions made during planning: --pdg-vocab/--mat-vocab default to 300 and len(MATERIAL_PROPERTIES); the consumed-keys report is scoped to conditioning/stage1_model/stage2_model only (train/meta are out of scope for a model-only build); the module tree prints every submodule at any depth. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 11:48:02 +02:00
lars	59eccbb5cb	Merge pull request 'Fix/issue 40' (#64) from fix/issue-40 into master ... Reviewed-on: #64 v0.3.2	2026-08-17 10:55:33 +02:00
lars	b42fa95d1a	Bump patch version to 0.3.2 ... Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 10:45:15 +02:00
lars	c1c4957e2f	Implement n_sec.mode = "stop_token" for the AR secondary decoder (gitea #40) ... Stage 2's autoregressive decoder still predicted multiplicity the v0.2 way: a one-shot n_sec_head classifier over conditioning alone, run before any secondary token existed, with the AR loop then always executing k_max slots and discarding the tail. This adds a real per-slot EOS mechanism instead: - Stage2Autoregressive gains a stop_head (build_stop_head=True) that predicts P(n_sec == k | prefix) at each slot, mutually exclusive with n_sec_head (n_sec.mode = "stop_token" builds no n_sec_head at all). - sample_secondaries_ar accepts n_sec_pred=None to drive generation off the stop head instead of a pre-resolved count: each row stops the first slot its stop logit fires (stage2_model.n_sec.stop_sampling = "greedy" — the default, threshold at 0 — or "sample", a Bernoulli draw), and the whole batch loop breaks once every row has stopped, so cost scales with the realized n_sec instead of a fixed k_max. Passing n_sec_pred explicitly (the scheduled-sampling self-sample path) is unchanged. - resolve_n_sec returns None for a stop-token decoder instead of raising; rollout.py/cli.py/validate.py now derive the realized count from sample_stage2's returned sec_valid (sec_valid.sum(-1)) after sampling, rather than resolving it up front — a no-op reordering under every other n_sec.mode, where sec_valid was already built from n_sec_pred. - Training: _stop_target_and_mask (giant/training/stage2_inputs.py) builds the per-slot target/mask (one slot wider than the existing token-content sec_mask, since the stop slot itself needs supervision) and StageTrainer._stop_loss trains it with masked BCE, gated on stop_head exactly like _n_sec_loss gates on n_sec_head. Wired into both the flow/ddpm trainer and the WGAN trainer (whose skip_g_step now also checks stop_head), weighted by the existing stage2_model.n_sec.lambda — the stop head replaces n_sec_head under this mode, so no new weight key. - validate_config now accepts stop_token (requires decoder="autoregressive" and n_sec.owner="stage2") instead of always rejecting it. Decisions made during planning: stop_sampling defaults to "greedy" for deterministic rollouts; the stop head reuses stage2_model.heads.n_sec's HeadConfig shape and stage2_model.n_sec.lambda's weight rather than adding new config keys, since the two heads never coexist. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 10:44:14 +02:00
lars	7bf0bea56a	Merge pull request 'Clamp analysis histogram bins before the i32 cast, not after (gitea #61)' (#63) from fix/issue-61 into master ... Reviewed-on: #63	2026-08-17 10:17:32 +02:00
lars	867a07da2b	Merge branch 'master' into fix/issue-61	2026-08-17 09:51:37 +02:00
lars	7514a4364f	Merge pull request 'Clip raw predicted log_mass in decode_secondaries (gitea #54)' (#62) from fix/issue-54 into master ... Reviewed-on: #62	2026-08-17 09:42:52 +02:00
lars	a746efb6e1	Clamp analysis histogram bins before the i32 cast, not after (gitea #61) ... _bin_expr in giant/analysis/reduce.py clipped the bin index to [0, nbins-1] only after casting it to Int32, so the clip never got a chance to run: a rollout step_length of 1.0725e10 mm against fixed edges [2.9e-5, 94.04] with 50 bins produces a raw index of ~5.7e9, which overflows i32 and fails the strict cast, killing the whole compute-one job. Same failure mode for +/-inf. Clamp in f64 first, then cast to Int32. NaN has no edge to clamp to, so it maps to null and is dropped in the two callers (hist1d, profile_partial) — matching what np.histogram does with NaN, and what profile_partial needs anyway since a null bin index would break its np.add.at. This reimplements commit 313373c, which fixed the same bug but landed on a branch (fix/rollout-negative-secondary-mass) that forked off a stale master and was never merged; reduce.py has since diverged enough that the original diff no longer applies cleanly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 09:40:09 +02:00
lars	bacc8763d0	Clip raw predicted log_mass in decode_secondaries (gitea #54) ... decode_secondaries inverted a secondary's raw predicted log_mass with inv_log_transform (exp(y) - eps) unclipped. log_mass is a raw regression output, not itself the result of log_transform, so it isn't guaranteed to land in the range that round-trips cleanly: too negative and exp(y) undershoots eps, making the result go slightly negative; too positive and exp(y) overflows float32 to inf. Either one crashes the next rollout step, since a track descended from that secondary feeds its mass back in as conditioning, and log_transform raises on a non-finite input. Clip log_mass to [log(_EPS), _LOG_MASS_MAX] before inverting, guaranteeing a finite, non-negative mass. _LOG_MASS_MAX=80.0 matches the value from the stale fix/rollout-negative-secondary-mass branch (comfortably below float32's ~88.7 overflow point, far beyond any physical particle mass a converged model would predict) — that branch had already implemented this fix but forked before gitea #35/#36 and couldn't be merged as-is, so this reimplements it fresh against current master and leaves the stale branch untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-17 09:31:37 +02:00
lars	ff435883ed	Merge pull request 'Let dwarf warm-cache take --config so it can't under-warm a config's cache keys (gitea #59)' (#60) from fix/issue-59 into master ... Reviewed-on: #60	2026-08-17 09:24:42 +02:00
lars	d25dfc0343	Let dwarf warm-cache take --config so it can't under-warm a config's cache keys (gitea #59) ... warm-cache built its config from DEFAULT_CONFIG with only a handful of flags overridable, so it had no way to express settings like stage2_model.particle_type.n_classes. configs/baseline.toml sets that to 32; warm-cache always warmed the pdg top-N map under the emb_dim default (16) instead, so a `giant train --config configs/baseline.toml` run silently missed the cache and repaid the full parquet scan warm-cache exists to avoid. warm-cache now accepts the same --config a training run takes and resolves every value run_setup_stage needs (val_fraction/seed, conditioning types, both stages' router, particle_type.n_classes, ...) from one gconfig.merge_cli_overrides + validate_config pass, exactly like giant train's own pipeline does — so warming and training are guaranteed to agree. Per user decision, --config is mutually exclusive with the individual --val-fraction/--seed/--particle-conditioning/ --material-conditioning/--router*/flags (rejected outright rather than silently layered on top), since a hardcoded CLI default clobbering an unset config value is the same failure mode one level down. Also drops a hardcoded stage2_model.router/k_max override that was a no-op against today's defaults but would have clobbered a config setting either one away from its default — same bug class. Adding validate_config surfaced that the existing test_warm_cache_router_process_warms_proc_map test was warming a router.type="process" + conditioning.particle.type="physical" (the CLI's old hardcoded default) combination that giant train's own validate_config would already reject as incompatible — fixed by passing --particle-conditioning embedding, which is what a working --router-type process run actually requires. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-17 08:54:14 +02:00
lars	a1ecf0df1d	Merge pull request 'Add configs/baseline.toml as the kept reference model' (#58) from add/baseline-config into master ... Reviewed-on: #58	2026-08-14 17:37:46 +02:00
lars	d858226294	Add configs/baseline.toml as the kept reference model ... A fixed comparison point for future architecture variants, so each experimental axis (routed trunk, WGAN generators, attention history, shared conditioning) is a single edit away from one known config. flow/flow autoregressive, hidden_dim 512 / 6 blocks per stage, physical conditioning, no router, 7.70M params. Chosen by ranking the five runs in analysis_runs/ by mean Jensen-Shannon divergence against the Geant4 reference: unrouted flow wins (0.172) over routed flow (0.197/0.200) and both WGAN runs (0.218/0.234), with the lead concentrated in per-event total deposited energy and the per-PDG marginals. batch_size 36864 is sized for one L40S on deepthought2 from a measured linear fit of this config's training step (reserved MiB = 0.9736 * bs + 115), giving ~36 GiB, 78% of the card. The comments record two measured facts that are easy to get wrong: WGAN is slower to *train* than flow (n_critic plus the gradient-penalty double-backward), its advantage being inference-only; and sample_secondaries_ar loops over all k_max slots unconditionally rather than short-circuiting on n_sec, which is what makes the autoregressive decoder the dominant cost on both axes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 17:36:26 +02:00
lars	4f092c4528	Merge pull request 'Give Stage1Model/Stage2OneShot/Stage2Autoregressive a shared StageModel base (gitea #39)' (#56) from fix/issue-39 into master ... Reviewed-on: #56 v0.3.1	2026-08-14 15:16:02 +02:00
lars	cc37a55183	Bump patch version to 0.3.1 ... Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-14 15:05:32 +02:00