perf: replace pandas with polars in the setup-stage scan
CI / Sync project version with tag (pull_request) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 1m6s
CI / Format (ruff format) (pull_request) Successful in 1m11s
CI / Type check (ty) (pull_request) Successful in 1m12s
CI / Tests (pull_request) Successful in 4m16s
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
CI / Publish package to Gitea package registry (pull_request) Has been skipped

giant.pipeline.run_setup_stage (used by both giant train and dwarf
warm-cache) previously opened and fully read each parquet file 4-6
separate times via pandas, with per-row Python loops padding the
secondary list columns on every chunk of the normalizer-fitting pass.

- giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list
  padding is now a single vectorized polars expression instead of a
  per-row Python loop (including a .iloc[i] loop for directions).
- giant/data/scan.py (new): a fused metadata scan answering the event
  index, pdg/material vocab, process counts, and pooled-pdg counts in
  one pass per file instead of one pass per section. Frequency-ranking
  ties are now an explicit (-count, first_seen) contract instead of an
  accident of pandas' value_counts iteration order.
- giant/pipeline.py: run_setup_stage restructured to consult the cache
  for every section first, then issue one combined scan request for
  whatever's missing.
- giant/geometry.py: ported the one remaining pandas groupby to polars.
- pyproject.toml: polars promoted to a core dependency, pandas moved
  to dev (only test fixtures still use it).
- giant/tools/profile_setup_scan.py (new): synthetic-data benchmark
  for this scan, mirroring profile_analysis_costs.py's pattern.

Also fixes a real deadlock this surfaced: DataLoader worker
subprocesses fork() on Linux, and polars' native thread pool doesn't
survive a fork — a worker touching polars after the parent already had
hangs instantly. giant/pipeline.py's train/val DataLoaders now use
multiprocessing_context="spawn" whenever num_workers>0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC
This commit is contained in:
2026-09-02 09:59:37 +02:00
parent deb9e8e7de
commit 7df1945384
10 changed files with 591 additions and 193 deletions
+23 -6
View File
@@ -108,12 +108,12 @@ def test_build_process_map_from_files_spans_multiple_files(tmp_path):
def test_build_process_map_from_files_tie_breaking_pins_first_seen_order(tmp_path):
"""When two processes end up with equal total counts, ranking falls back
to whichever was accumulated first (`sorted(..., reverse=True)` is stable,
and `counts` is built in file/row-scan order) — this is implementation-
defined, not a documented contract, so pin it explicitly: a future
rewrite (e.g. a polars-based single-scan) that ties differently would
silently reshuffle which processes get their own expert slot across a
retrain, and this test is what should catch that."""
to whichever was scanned first — file order, then row order within a
file (`giant.data.scan`'s `first_seen` ordinal, ranked by
`giant.data.loader._topn_plus_other_map`'s `(-count, first_seen)` key).
This is an explicit, documented contract (not an accident of iteration
order), pinned here so a future change to the ranking can't silently
reshuffle which processes get their own expert slot across a retrain."""
path = tmp_path / "a.parquet"
pd.DataFrame({"process": ["compt", "phot", "compt", "phot"]}).to_parquet(path)
@@ -233,6 +233,23 @@ def test_build_pdg_topn_map_from_files_pools_primary_and_secondary_pdg(tmp_path)
assert m.class_counts == {0: 11, 1: 5}
def test_build_pdg_topn_map_from_files_pooled_tie_breaks_by_row_position(tmp_path):
"""Pooled pdg counting merges the primary `pdg` column and the exploded
`sec_pdg_list` column via one `group_by` over both (see
`giant.data.scan._pooled_pdg_lazy`), keyed by row position regardless of
which role (primary or secondary) a code was seen in — not "all
primaries before all secondaries" the way a two-pass accumulation would.
11 (primary, row 0), 33 (primary, row 1) and 22 (secondary, row 1) all
end up with count 1; 11's strictly earlier row wins the tie over both,
whatever order 33/22 (tied with each other, same row) land in."""
path = tmp_path / "a.parquet"
pd.DataFrame({"pdg": [11, 33], "sec_pdg_list": [[], [22]]}).to_parquet(path)
m = build_pdg_topn_map_from_files([path], n_classes=4)
assert m.class_map[11] == 0
def test_build_pdg_topn_map_from_files_missing_sec_pdg_list_column(tmp_path):
"""Files predating the parent->child join have no sec_pdg_list column —
must not raise, just count the primary pdg column alone."""