The v0.3.0 pivot exists because the 2026-08-03 WGAN rollout benchmark
produced zero photon secondaries and ~4M hallucinated antineutrinos —
even with a correctly-sized top-N species vocabulary (gitea #29), plain
cross-entropy over a class distribution spanning orders of magnitude
still under-predicts rare-but-physical species.
stage2_model.particle_type.class_weighting = "none" | "inverse_freq"
(default "none", fully back-compat) weights the stage-2 type head's CE
loss (FlowDDPMStageTrainer._type_loss) by inverse class frequency,
normalized to mean 1 so switching it on doesn't rescale the type loss
against particle_type.lambda / the generator loss it's summed with.
The per-class counts the weighting needs don't already exist despite the
issue's premise: _topn_plus_other_map (giant/data/loader.py) previously
kept counts only for keys folded into "other", dropping the kept classes'
counts on the floor. TopNMap now carries class_counts (index -> count),
round-tripped through the setup-cache sidecar (format version bumped
3->4, since existing sidecars have none) and through checkpoints
(tolerantly — a pre-#44 checkpoint decodes to {}, since only training-time
loss weighting reads it, not inference).
Decisions made during planning (with the user): dropped "effective_num"
from the issue's proposed three-way enum (no beta hyperparameter to
design around) — final domain is "none" | "inverse_freq". Weights are
mean-1-normalized. validate_config rejects class_weighting != "none"
combined with particle_type.target != "onehot" or
stage2_model.generator == "wgan" (both have no class CE to weight),
following the #28/#30 dead-key-must-not-go-silent convention. Branch
fix/issue-44 off master.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`scripts` was published as a top-level distribution package, colliding
with one of the most generic names in the Python ecosystem and
shadowable by a stray scripts/ dir on the portal machines' shared
/work/lbogner. Move it under the giant namespace; the dwarf command
name is unchanged, only the Python import path and file location move.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The design doc and its followups doc are no longer needed as a live
reference now that the v0.3.0 redesign is implemented — comments and
docstrings across the codebase cited it extensively (file path, "design
doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design
rationale. Removed docs/ and edited every citing comment/docstring to
drop the now-dangling reference while keeping the substantive
explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its
trailing pointer to the deleted file.
Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or
"§N.N" references (repo-wide grep); ruff and ty clean; full test suite
on the heaviest-touched modules (network, sample, rollout, migration,
config, train) passes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Works through docs/v0.3.0-followups.md item by item, closing the gap
between the design doc and the shipped v0.3.0-stage2-autoregressive code:
1. validate.py: 7-tuple batch unpacking, sample_stage1/sample_stage2
dispatch, stage-2 particle-type-class marginal.
2. Stage-prefixed --stage1-*/--stage2-* CLI flags for train/new-run.
3. Thread stage2_model.k_max through loader/transforms/dataset/pipeline/
train instead of the hardcoded K_MAX constant.
4. Mixed conditioning.particle.type / conditioning.material.type support
end-to-end (data pipeline + dwarf warm-cache).
5. conditioning.share_stages = true: one shared ConditionEncoder instance
across both stages.
6. stage2_model.generator = "ddpm" formally deferred into design doc §11.2
(was silently unimplemented).
7. giant predict/rollout: implement conditioning.*.type = "onehot" via the
checkpoint's saved pdg_topn_map/mat_topn_map.
8. network.py's checkpoint-path model_config migration now fails loudly on
non-zero legacy expert_hidden_dim/expert_n_blocks, matching config.py's
TOML-load path (§4.2).
9. validate_config now rejects stage2_model.n_sec.mode = "truth" for a
rollout-capable checkpoint (§9).
Also cleared all pre-existing `ty check` noise (44 -> 0 diagnostics),
mostly a test-helper dict-unpack pattern that made every unrelated
constructor keyword look like a type error.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Builds the shared top-N-plus-other PDG/material maps (pooling both primary
and secondary occurrences for PDG, directly targeting the meeting's
species-collapse failure mode) and wires up conditioning.{particle,material}
= "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding")
end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm)
vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding
CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte
unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact.
giant predict/rollout fail loudly on a onehot/embedding checkpoint until
full decode support lands in step 6.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- log_transform / _validate_unit_pre_dir now raise on non-finite input
instead of letting a NaN row silently poison the persisted normalizer
cache (norm < 1e-6 was always False for NaN, so the existing guard
never caught it).
- encode_secondaries warns when a row's secondary energies cumulatively
exceed e_sec, instead of silently saturating the overflowing slot's
stick-breaking logit via the _EPS floor.
- EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding
two files' event ids together (reintroducing train/val leakage).
- make_event_split(val_fraction=0.0) now actually holds out nothing,
instead of always forcing at least 1 validation event.
- setup_cache.save() is now serialized with a flock, since two
concurrent writers (a real scenario on this repo's shared
portal/condor machines) could otherwise race and silently drop one
writer's freshly-computed cache section.
- Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation
singularity in _rodrigues_axis, which is real but inherent to any
single-valued local-frame convention.
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py),
and a job's event_id numbering always restarts from 0 — so loading
multiple files together (a directory or .manifest) let same-numbered
events from different files collapse into one during the event index
scan and train/val split, corrupting both. Every per-file event_id now
gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py),
threaded through the setup-cache event index, the streaming dataset,
and predict/rollout seeding. Bumps the setup-cache format version so
stale sidecars computed pre-fix are invalidated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Brings in the rollout-validation fixes developed alongside Phase 2
(exact e_sec budget rescaling in decode_secondaries, filtering
synthetic termination rows out of load_rollout_vs_truth, Tier 4 truth
overlay, --energy-gev support in dwarf make-root) and reconciles them
with this branch's mixture-of-experts routing work: build_features/
build_models/dataset plumbing keep the ProcessRouter's proc_map/
proc_idx threading, and create_root_files.py's job_seed folds in both
the per-job seed derivation and the new energy_gev component.
Routes on the physics process (Compton, phot, brems, ...) that ends a
step, supervised by a small classifier since process is a post-step
outcome unobservable at gate time. Threads a process label end-to-end
through the data pipeline (loader, build_features, dataset batches,
training loss/checkpointing) alongside the existing EnergyRouter.
Introduces raw/<kind>/<gen>/<detector>/shard-NNN.root and
processed/<kind>/<gen>/<schema>/<detector>/shard-NNN.parquet as the dataset
convention, plus scripts to operate on it: migrate_geant_steps.py for the
one-time move into this layout, bump_dataset_version.py to cut new
gen/schema versions with a logged reason, steps_to_parquet_parallel.py to
convert ROOT shards to parquet in parallel and place them correctly, and
create_root_files.py to generate new ROOT shards via a minicalosim
executable. The loader gains .manifest file support so pools/ (train/dev/
holdout shard lists) can be passed straight to `giant train`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replaces the independent log_delta_e/log_edep targets with 2 additive-log-ratio
coordinates over the deposit/secondary/post-energy simplex (fractions of pre_E
summing to 1), so edep + e_sec + post_E == pre_E holds by construction after
decoding (softmax) rather than being learned approximately. Requires e_sec
(secondary energy) as a new conditioning input and a steps_to_parquet.py pass
to derive it from child track first-step energies.
step_length already encodes |post_pos - pre_pos| by definition, so a raw
post_pos target would duplicate that magnitude and could drift inconsistent
with step_length during sampling. Instead add travel_dir, a unit vector
(local frame) giving only the direction of pre_pos->post_pos; post_pos is
reconstructed at inference as pre_pos + step_length * travel_dir, keeping
the two self-consistent. Target grows from 6D to 9D.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
material is a string literal (e.g. "G4_PbWO4"), not an integer. Store as
object array and key mat_map on str throughout loader and transforms.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- iter_cond_chunks: column-projected row-group streaming; post-step
variables are never read from disk during inference
- build_cond_features: assembles conditioning arrays without any target
or post-step fields
- inv_local_frame_rotation: Rodrigues R^T (negative angle) to rotate
predicted post_dir back from local frame to world frame
- giant predict: loads checkpoint, streams input, runs flow matching
sampler, inverse-normalises and inverse-rotates outputs, writes
predictions incrementally as parquet via PyArrow ParquetWriter
- train now saves model_config in checkpoint so predict can reconstruct
the architecture without extra CLI flags
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Streaming pipeline: row-group-level parquet reading (PyArrow) so
large files never fully land in RAM; Welford online algorithm for
normalizer fitting; StreamingStepsDataset with shuffle buffer and
multi-worker file striping; event-ID scan and vocab scan via cheap
single-column reads
- giant/cli.py: typer-based CLI with `giant train` subcommand, mirroring
scripts/train.py; --shuffle-buffer flag for RAM control
- pyproject.toml: add typer>=0.12 dependency and giant entry point
- train.py: replace len(loader.dataset) with local counters (compatible
with IterableDataset which has no __len__)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lars