11 Commits

Author	SHA1	Message	Date
lars	7df1945384	perf: replace pandas with polars in the setup-stage scan ... giant.pipeline.run_setup_stage (used by both giant train and dwarf warm-cache) previously opened and fully read each parquet file 4-6 separate times via pandas, with per-row Python loops padding the secondary list columns on every chunk of the normalizer-fitting pass. - giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list padding is now a single vectorized polars expression instead of a per-row Python loop (including a .iloc[i] loop for directions). - giant/data/scan.py (new): a fused metadata scan answering the event index, pdg/material vocab, process counts, and pooled-pdg counts in one pass per file instead of one pass per section. Frequency-ranking ties are now an explicit (-count, first_seen) contract instead of an accident of pandas' value_counts iteration order. - giant/pipeline.py: run_setup_stage restructured to consult the cache for every section first, then issue one combined scan request for whatever's missing. - giant/geometry.py: ported the one remaining pandas groupby to polars. - pyproject.toml: polars promoted to a core dependency, pandas moved to dev (only test fixtures still use it). - giant/tools/profile_setup_scan.py (new): synthetic-data benchmark for this scan, mirroring profile_analysis_costs.py's pattern. Also fixes a real deadlock this surfaced: DataLoader worker subprocesses fork() on Linux, and polars' native thread pool doesn't survive a fork — a worker touching polars after the parent already had hangs instantly. giant/pipeline.py's train/val DataLoaders now use multiprocessing_context="spawn" whenever num_workers>0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC	2026-09-02 09:59:37 +02:00
lars	fce47b128c	Add class-balanced secondary particle-type loss (gitea #44) ... The v0.3.0 pivot exists because the 2026-08-03 WGAN rollout benchmark produced zero photon secondaries and ~4M hallucinated antineutrinos — even with a correctly-sized top-N species vocabulary (gitea #29), plain cross-entropy over a class distribution spanning orders of magnitude still under-predicts rare-but-physical species. stage2_model.particle_type.class_weighting = "none" | "inverse_freq" (default "none", fully back-compat) weights the stage-2 type head's CE loss (FlowDDPMStageTrainer._type_loss) by inverse class frequency, normalized to mean 1 so switching it on doesn't rescale the type loss against particle_type.lambda / the generator loss it's summed with. The per-class counts the weighting needs don't already exist despite the issue's premise: _topn_plus_other_map (giant/data/loader.py) previously kept counts only for keys folded into "other", dropping the kept classes' counts on the floor. TopNMap now carries class_counts (index -> count), round-tripped through the setup-cache sidecar (format version bumped 3->4, since existing sidecars have none) and through checkpoints (tolerantly — a pre-#44 checkpoint decodes to {}, since only training-time loss weighting reads it, not inference). Decisions made during planning (with the user): dropped "effective_num" from the issue's proposed three-way enum (no beta hyperparameter to design around) — final domain is "none" | "inverse_freq". Weights are mean-1-normalized. validate_config rejects class_weighting != "none" combined with particle_type.target != "onehot" or stage2_model.generator == "wgan" (both have no class CE to weight), following the #28/#30 dead-key-must-not-go-silent convention. Branch fix/issue-44 off master. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-17 23:02:50 +02:00
lars	55332db67a	Bump ruff line-length to 120 and reformat ... Rejoins lines that only wrapped because they exceeded the old 88-char limit; ruff check and the full test suite (725 passed) are unaffected.	2026-08-12 13:33:09 +02:00
lars	878e9ddca3	Delete docs/v0.3.0-design.md and strip all references to it ... The design doc and its followups doc are no longer needed as a live reference now that the v0.3.0 redesign is implemented — comments and docstrings across the codebase cited it extensively (file path, "design doc §X.Y", "decision N", or bare "§X.Y" section numbers) as design rationale. Removed docs/ and edited every citing comment/docstring to drop the now-dangling reference while keeping the substantive explanation next to it. CLAUDE.md's v0.3.0 roadmap bullet loses its trailing pointer to the deleted file. Verified: no remaining "docs/v0.3.0", "design doc", "decision N", or "§N.N" references (repo-wide grep); ruff and ty clean; full test suite on the heaviest-touched modules (network, sample, rollout, migration, config, train) passes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-10 11:19:02 +02:00
lars	4fc15ecdfc	v0.3.0 step 4: type map + particle_type.target = "onehot"/"embedding" ... Builds the shared top-N-plus-other PDG/material maps (pooling both primary and secondary occurrences for PDG, directly targeting the meeting's species-collapse failure mode) and wires up conditioning.{particle,material} = "onehot" plus stage2_model.particle_type.target in ("onehot", "embedding") end-to-end: setup-cache persistence, Stage2OneShot's type_head (flow/ddpm) vs. folded+ST-Gumbel-relaxed adversarial slice (wgan), and the corresponding CE/MSE training losses. particle_type.target = "physical" stays byte-for-byte unchanged, keeping the v0.2 migration shim's bit-identical guarantee intact. giant predict/rollout fail loudly on a onehot/embedding checkpoint until full decode support lands in step 6. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-06 15:43:48 +02:00
lars	74343d3e48	Add data-integrity guards against silent NaN/Inf propagation and races ... - log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-03 13:47:29 +02:00
lars	b944bba8fb	Offset event_id per file to avoid cross-file collisions ... Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-30 11:36:48 +02:00
lars	47a6c9db1f	Add regression coverage for vocab/process index-map builders ... build_index_maps, build_index_maps_from_files, and (mostly) build_process_map_from_files had no test pinning their sort order, tie-breaking, or cross-file union behavior — all load-bearing for a trained checkpoint's vocabulary, and all at risk of silently changing under a future single-pass (pyarrow/polars) rewrite of the setup-stage scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion codes), negative PDG codes, dedup/bijective indices, file-order independence, and process-map tie-breaking/boundary conditions (n_experts=1, fewer processes than experts, 3-file partial overlap). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-29 13:45:33 +02:00
lars	05d5dee606	Apply ruff format after merging phase2-secondary-prediction ... The merged proc_idx/proc_map plumbing wasn't run through ruff format before merging; reflow only, no logic changes.	2026-07-15 10:00:36 +02:00
lars	0b3ece52ed	Add ProcessRouter for physics-process-based expert gating ... Routes on the physics process (Compton, phot, brems, ...) that ends a step, supervised by a small classifier since process is a post-step outcome unobservable at gate time. Threads a process label end-to-end through the data pipeline (loader, build_features, dataset batches, training loss/checkpointing) alongside the existing EnergyRouter.	2026-07-08 16:15:09 +02:00
lars	320365606a	Add tooling for a versioned geant_steps dataset layout ... Introduces raw/<kind>/<gen>/<detector>/shard-NNN.root and processed/<kind>/<gen>/<schema>/<detector>/shard-NNN.parquet as the dataset convention, plus scripts to operate on it: migrate_geant_steps.py for the one-time move into this layout, bump_dataset_version.py to cut new gen/schema versions with a logged reason, steps_to_parquet_parallel.py to convert ROOT shards to parquet in parallel and place them correctly, and create_root_files.py to generate new ROOT shards via a minicalosim executable. The loader gains .manifest file support so pools/ (train/dev/ holdout shard lists) can be passed straight to `giant train`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-06-25 16:46:48 +02:00