The streaming update re-derived two full (B, F) arrays from the
running mean (once before updating it, once after) plus an elementwise
product — five passes over each chunk and three temporary arrays, to
maintain a mean/variance that's tiny in width (COND_DIM=15 at most).
Reformulate as Chan/Golub/LeVeque's parallel-variance algorithm:
compute the chunk's own local mean/M2 (independent of the running
state) and merge it in with an O(F) combination formula. Same
streaming interface and output (identical to ~1e-14, float64 rounding
noise), ~40% faster per update() call on a benchmark chunk.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
build_index_maps, build_index_maps_from_files, and (mostly)
build_process_map_from_files had no test pinning their sort order,
tie-breaking, or cross-file union behavior — all load-bearing for a
trained checkpoint's vocabulary, and all at risk of silently changing
under a future single-pass (pyarrow/polars) rewrite of the setup-stage
scan. Add tests for numeric-vs-lexicographic PDG sort (nuclear/ion
codes), negative PDG codes, dedup/bijective indices, file-order
independence, and process-map tie-breaking/boundary conditions
(n_experts=1, fewer processes than experts, 3-file partial overlap).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fits normalizers over multi-hundred-million-row datasets, so the setup
pass's per-row Python overhead compounds fast: encode_secondaries
recomputed an O(K) prefix sum from scratch on every one of its 15
stick-breaking iterations, np.isin re-sorted the full train-event-id
array on every chunk, and pdg/material/process index lookups ran a
Python dict lookup per row. The normalizer-fit pass also computed
encode_secondaries's stick-logit and direction-rotation blocks in full
even though it only ever reads the mass/charge columns.
Replace the prefix-sum recompute with a single np.cumsum, add a
sorted_membership helper (searchsorted-based) in place of np.isin at
both the setup-pass and per-epoch call sites, vectorize the index
lookups via _vectorized_map_lookup, and add an opt-in phys_only path
so the setup pass skips the stick-breaking/rotation work it discards
anyway. All four changes are output-identical performance refactors,
backed by new unit tests plus the existing suite.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars