perf: replace pandas with polars in the setup-stage scan
CI / Sync project version with tag (pull_request) Has been skipped
CI / Lint (ruff check) (pull_request) Successful in 1m6s
CI / Format (ruff format) (pull_request) Successful in 1m11s
CI / Type check (ty) (pull_request) Successful in 1m12s
CI / Tests (pull_request) Successful in 4m16s
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
CI / Publish package to Gitea package registry (pull_request) Has been skipped

giant.pipeline.run_setup_stage (used by both giant train and dwarf
warm-cache) previously opened and fully read each parquet file 4-6
separate times via pandas, with per-row Python loops padding the
secondary list columns on every chunk of the normalizer-fitting pass.

- giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list
  padding is now a single vectorized polars expression instead of a
  per-row Python loop (including a .iloc[i] loop for directions).
- giant/data/scan.py (new): a fused metadata scan answering the event
  index, pdg/material vocab, process counts, and pooled-pdg counts in
  one pass per file instead of one pass per section. Frequency-ranking
  ties are now an explicit (-count, first_seen) contract instead of an
  accident of pandas' value_counts iteration order.
- giant/pipeline.py: run_setup_stage restructured to consult the cache
  for every section first, then issue one combined scan request for
  whatever's missing.
- giant/geometry.py: ported the one remaining pandas groupby to polars.
- pyproject.toml: polars promoted to a core dependency, pandas moved
  to dev (only test fixtures still use it).
- giant/tools/profile_setup_scan.py (new): synthetic-data benchmark
  for this scan, mirroring profile_analysis_costs.py's pattern.

Also fixes a real deadlock this surfaced: DataLoader worker
subprocesses fork() on Linux, and polars' native thread pool doesn't
survive a fork — a worker touching polars after the parent already had
hangs instantly. giant/pipeline.py's train/val DataLoaders now use
multiprocessing_context="spawn" whenever num_workers>0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC
This commit is contained in:
2026-09-02 09:59:37 +02:00
parent deb9e8e7de
commit 7df1945384
10 changed files with 591 additions and 193 deletions
+5 -1
View File
@@ -6,7 +6,7 @@ readme = "README.md"
requires-python = ">=3.12"
dependencies = [
"numpy>=1.26,<3",
"pandas>=2.2,<4",
"polars>=1.0,<2",
"pyarrow>=16,<25",
"tqdm>=4.60,<5",
"typer>=0.12,<1",
@@ -28,6 +28,10 @@ dev = [
"ty>=0.0.50,<0.1",
"bump-my-version>=1.2,<2",
"git-cliff>=2,<3",
# Only used by test fixtures (writing small parquet files) — not a
# runtime dependency of giant itself since the pandas -> polars
# data-loading rewrite.
"pandas>=2.2,<4",
"giant[convert,analysis,geometry,wandb]",
]
geometry = [