perf: replace pandas with polars in the setup-stage scan
CI / Sync project version with tag (pull_request) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 1m6s
CI / Format (ruff format) (pull_request) Successful in 1m11s
CI / Type check (ty) (pull_request) Successful in 1m12s
CI / Tests (pull_request) Successful in 4m16s
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
CI / Publish package to Gitea package registry (pull_request) Has been skipped
CI / Sync project version with tag (pull_request) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 1m6s
CI / Format (ruff format) (pull_request) Successful in 1m11s
CI / Type check (ty) (pull_request) Successful in 1m12s
CI / Tests (pull_request) Successful in 4m16s
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
CI / Publish package to Gitea package registry (pull_request) Has been skipped
giant.pipeline.run_setup_stage (used by both giant train and dwarf warm-cache) previously opened and fully read each parquet file 4-6 separate times via pandas, with per-row Python loops padding the secondary list columns on every chunk of the normalizer-fitting pass. - giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list padding is now a single vectorized polars expression instead of a per-row Python loop (including a .iloc[i] loop for directions). - giant/data/scan.py (new): a fused metadata scan answering the event index, pdg/material vocab, process counts, and pooled-pdg counts in one pass per file instead of one pass per section. Frequency-ranking ties are now an explicit (-count, first_seen) contract instead of an accident of pandas' value_counts iteration order. - giant/pipeline.py: run_setup_stage restructured to consult the cache for every section first, then issue one combined scan request for whatever's missing. - giant/geometry.py: ported the one remaining pandas groupby to polars. - pyproject.toml: polars promoted to a core dependency, pandas moved to dev (only test fixtures still use it). - giant/tools/profile_setup_scan.py (new): synthetic-data benchmark for this scan, mirroring profile_analysis_costs.py's pattern. Also fixes a real deadlock this surfaced: DataLoader worker subprocesses fork() on Linux, and polars' native thread pool doesn't survive a fork — a worker touching polars after the parent already had hangs instantly. giant/pipeline.py's train/val DataLoaders now use multiprocessing_context="spawn" whenever num_workers>0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC
This commit is contained in:
+23
-6
@@ -108,12 +108,12 @@ def test_build_process_map_from_files_spans_multiple_files(tmp_path):
|
||||
|
||||
def test_build_process_map_from_files_tie_breaking_pins_first_seen_order(tmp_path):
|
||||
"""When two processes end up with equal total counts, ranking falls back
|
||||
to whichever was accumulated first (`sorted(..., reverse=True)` is stable,
|
||||
and `counts` is built in file/row-scan order) — this is implementation-
|
||||
defined, not a documented contract, so pin it explicitly: a future
|
||||
rewrite (e.g. a polars-based single-scan) that ties differently would
|
||||
silently reshuffle which processes get their own expert slot across a
|
||||
retrain, and this test is what should catch that."""
|
||||
to whichever was scanned first — file order, then row order within a
|
||||
file (`giant.data.scan`'s `first_seen` ordinal, ranked by
|
||||
`giant.data.loader._topn_plus_other_map`'s `(-count, first_seen)` key).
|
||||
This is an explicit, documented contract (not an accident of iteration
|
||||
order), pinned here so a future change to the ranking can't silently
|
||||
reshuffle which processes get their own expert slot across a retrain."""
|
||||
path = tmp_path / "a.parquet"
|
||||
pd.DataFrame({"process": ["compt", "phot", "compt", "phot"]}).to_parquet(path)
|
||||
|
||||
@@ -233,6 +233,23 @@ def test_build_pdg_topn_map_from_files_pools_primary_and_secondary_pdg(tmp_path)
|
||||
assert m.class_counts == {0: 11, 1: 5}
|
||||
|
||||
|
||||
def test_build_pdg_topn_map_from_files_pooled_tie_breaks_by_row_position(tmp_path):
|
||||
"""Pooled pdg counting merges the primary `pdg` column and the exploded
|
||||
`sec_pdg_list` column via one `group_by` over both (see
|
||||
`giant.data.scan._pooled_pdg_lazy`), keyed by row position regardless of
|
||||
which role (primary or secondary) a code was seen in — not "all
|
||||
primaries before all secondaries" the way a two-pass accumulation would.
|
||||
11 (primary, row 0), 33 (primary, row 1) and 22 (secondary, row 1) all
|
||||
end up with count 1; 11's strictly earlier row wins the tie over both,
|
||||
whatever order 33/22 (tied with each other, same row) land in."""
|
||||
path = tmp_path / "a.parquet"
|
||||
pd.DataFrame({"pdg": [11, 33], "sec_pdg_list": [[], [22]]}).to_parquet(path)
|
||||
|
||||
m = build_pdg_topn_map_from_files([path], n_classes=4)
|
||||
|
||||
assert m.class_map[11] == 0
|
||||
|
||||
|
||||
def test_build_pdg_topn_map_from_files_missing_sec_pdg_list_column(tmp_path):
|
||||
"""Files predating the parent->child join have no sec_pdg_list column —
|
||||
must not raise, just count the primary pdg column alone."""
|
||||
|
||||
Reference in New Issue
Block a user