build_index_maps, build_index_maps_from_files, and (mostly)
build_process_map_from_files had no test pinning their sort order,
tie-breaking, or cross-file union behavior — all load-bearing for a
trained checkpoint's vocabulary, and all at risk of silently changing
under a future single-pass (pyarrow/polars) rewrite of the setup-stage
scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion
codes), negative PDG codes, dedup/bijective indices, file-order
independence, and process-map tie-breaking/boundary conditions
(n_experts=1, fewer processes than experts, 3-file partial overlap).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Routes on the physics process (Compton, phot, brems, ...) that ends a
step, supervised by a small classifier since process is a post-step
outcome unobservable at gate time. Threads a process label end-to-end
through the data pipeline (loader, build_features, dataset batches,
training loss/checkpointing) alongside the existing EnergyRouter.
Introduces raw/<kind>/<gen>/<detector>/shard-NNN.root and
processed/<kind>/<gen>/<schema>/<detector>/shard-NNN.parquet as the dataset
convention, plus scripts to operate on it: migrate_geant_steps.py for the
one-time move into this layout, bump_dataset_version.py to cut new
gen/schema versions with a logged reason, steps_to_parquet_parallel.py to
convert ROOT shards to parquet in parallel and place them correctly, and
create_root_files.py to generate new ROOT shards via a minicalosim
executable. The loader gains .manifest file support so pools/ (train/dev/
holdout shard lists) can be passed straight to `giant train`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lars