lars/giant
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

perf: replace pandas with polars in the setup-stage scan #93

Merged
lars merged 1 commits from warm-cache-polars-scan into master 2026-09-02 10:07:07 +02:00
Conversation 0 Commits 1 Files Changed 10
+591 -193
lars commented 2026-09-02 10:00:58 +02:00
Owner
Copy Link
Copy Source

giant.pipeline.run_setup_stage (used by both giant train and dwarf
warm-cache) previously opened and fully read each parquet file 4-6
separate times via pandas, with per-row Python loops padding the
secondary list columns on every chunk of the normalizer-fitting pass.

  • giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list
    padding is now a single vectorized polars expression instead of a
    per-row Python loop (including a .iloc[i] loop for directions).
  • giant/data/scan.py (new): a fused metadata scan answering the event
    index, pdg/material vocab, process counts, and pooled-pdg counts in
    one pass per file instead of one pass per section. Frequency-ranking
    ties are now an explicit (-count, first_seen) contract instead of an
    accident of pandas' value_counts iteration order.
  • giant/pipeline.py: run_setup_stage restructured to consult the cache
    for every section first, then issue one combined scan request for
    whatever's missing.
  • giant/geometry.py: ported the one remaining pandas groupby to polars.
  • pyproject.toml: polars promoted to a core dependency, pandas moved
    to dev (only test fixtures still use it).
  • giant/tools/profile_setup_scan.py (new): synthetic-data benchmark
    for this scan, mirroring profile_analysis_costs.py's pattern.

Also fixes a real deadlock this surfaced: DataLoader worker
subprocesses fork() on Linux, and polars' native thread pool doesn't
survive a fork — a worker touching polars after the parent already had
hangs instantly. giant/pipeline.py's train/val DataLoaders now use
multiprocessing_context="spawn" whenever num_workers>0.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC

giant.pipeline.run_setup_stage (used by both giant train and dwarf warm-cache) previously opened and fully read each parquet file 4-6 separate times via pandas, with per-row Python loops padding the secondary list columns on every chunk of the normalizer-fitting pass. - giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list padding is now a single vectorized polars expression instead of a per-row Python loop (including a .iloc[i] loop for directions). - giant/data/scan.py (new): a fused metadata scan answering the event index, pdg/material vocab, process counts, and pooled-pdg counts in one pass per file instead of one pass per section. Frequency-ranking ties are now an explicit (-count, first_seen) contract instead of an accident of pandas' value_counts iteration order. - giant/pipeline.py: run_setup_stage restructured to consult the cache for every section first, then issue one combined scan request for whatever's missing. - giant/geometry.py: ported the one remaining pandas groupby to polars. - pyproject.toml: polars promoted to a core dependency, pandas moved to dev (only test fixtures still use it). - giant/tools/profile_setup_scan.py (new): synthetic-data benchmark for this scan, mirroring profile_analysis_costs.py's pattern. Also fixes a real deadlock this surfaced: DataLoader worker subprocesses fork() on Linux, and polars' native thread pool doesn't survive a fork — a worker touching polars after the parent already had hangs instantly. giant/pipeline.py's train/val DataLoaders now use multiprocessing_context="spawn" whenever num_workers>0. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC
lars added 1 commit 2026-09-02 10:00:59 +02:00
perf: replace pandas with polars in the setup-stage scan
CI / Sync project version with tag (pull_request) Has been skipped
Details
CI / Lint (ruff check) (pull_request) Successful in 1m6s
Details
CI / Format (ruff format) (pull_request) Successful in 1m11s
Details
CI / Type check (ty) (pull_request) Successful in 1m12s
Details
CI / Tests (pull_request) Successful in 4m16s
Details
CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped
Details
CI / Publish package to Gitea package registry (pull_request) Has been skipped
Details
7df1945384 
giant.pipeline.run_setup_stage (used by both giant train and dwarf
warm-cache) previously opened and fully read each parquet file 4-6
separate times via pandas, with per-row Python loops padding the
secondary list columns on every chunk of the normalizer-fitting pass.

- giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list
  padding is now a single vectorized polars expression instead of a
  per-row Python loop (including a .iloc[i] loop for directions).
- giant/data/scan.py (new): a fused metadata scan answering the event
  index, pdg/material vocab, process counts, and pooled-pdg counts in
  one pass per file instead of one pass per section. Frequency-ranking
  ties are now an explicit (-count, first_seen) contract instead of an
  accident of pandas' value_counts iteration order.
- giant/pipeline.py: run_setup_stage restructured to consult the cache
  for every section first, then issue one combined scan request for
  whatever's missing.
- giant/geometry.py: ported the one remaining pandas groupby to polars.
- pyproject.toml: polars promoted to a core dependency, pandas moved
  to dev (only test fixtures still use it).
- giant/tools/profile_setup_scan.py (new): synthetic-data benchmark
  for this scan, mirroring profile_analysis_costs.py's pattern.

Also fixes a real deadlock this surfaced: DataLoader worker
subprocesses fork() on Linux, and polars' native thread pool doesn't
survive a fork — a worker touching polars after the parent already had
hangs instantly. giant/pipeline.py's train/val DataLoaders now use
multiprocessing_context="spawn" whenever num_workers>0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC
lars merged commit 674f7254cd into master 2026-09-02 10:07:07 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
architecture
bug
Silent/wrong behavior, no crash
 experiment
feature
New feature matching the v0.3 redesign
 investigation
modularity
Making the network genuinely pluggable
 performance
phase2
validation
wontfix
Not being fixed right now
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
lars (Lars Bogner)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lars/giant#93
Delete Branch "warm-cache-polars-scan"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?