Rewrite analysis module as a lean, fully-streaming pipeline #12

2026-07-17T12:34:00+02:00

lbogner commented

2026-07-17 12:34:00 +02:00

(Migrated from gitlab.etp.kit.edu)

Replace giant/analysis.py's dual numpy-SampleCollection + polars paths with a
single polars-streaming implementation that produces the validation notebook's
plots directly from a giant predict --coord local parquet, sized for files
larger than RAM.

Drop the numpy SampleCollection path (load_predicted_local, marginal_table,
correlation_matrices, direction_alignment, constraint_report, plot_kl_bars)
and the rollout observables; the 5 remaining plotters now take a parquet
path / LazyFrame and stream internally.
Rewrite compute_event_observables_pl to aggregate in parallel streaming
polars (post-pos reconstruction as expressions) instead of a serial
pyarrow-batch + numpy loop, fixing a pre-existing OOM (holistic median +
323M-row join in the bin-edge sizing). Medians are approximated from a
streaming log-bin histogram with within-bin interpolation.
Keep every full-file scan narrow (few columns): on a file larger than RAM,
peak mmap memory, not scan count, is the binding constraint. Marginals run
one dim at a time (~15GB peak) rather than a combined all-dims pass (OOM).
Update analysis/validation.ipynb to the path-based API; delete the
analysis/export_.py and compare_ode_steps_.py one-off scripts.
Rewrite tests/test_analysis.py around parquet fixtures with an inline numpy
oracle; add correlation/streaming-plotter and approx-median coverage.

Verified end-to-end on the 32GB predict file: full notebook completes at
~25GB peak (no OOM); event rollup runs at ~13 cores.

Co-Authored-By: Claude Opus 4.8 noreply@anthropic.com

Replace giant/analysis.py's dual numpy-SampleCollection + polars paths with a single polars-streaming implementation that produces the validation notebook's plots directly from a `giant predict --coord local` parquet, sized for files larger than RAM. - Drop the numpy SampleCollection path (load_predicted_local, marginal_table, correlation_matrices, direction_alignment, constraint_report, plot_kl_bars) and the rollout observables; the 5 remaining plotters now take a parquet path / LazyFrame and stream internally. - Rewrite compute_event_observables_pl to aggregate in parallel streaming polars (post-pos reconstruction as expressions) instead of a serial pyarrow-batch + numpy loop, fixing a pre-existing OOM (holistic median + 323M-row join in the bin-edge sizing). Medians are approximated from a streaming log-bin histogram with within-bin interpolation. - Keep every full-file scan narrow (few columns): on a file larger than RAM, peak mmap memory, not scan count, is the binding constraint. Marginals run one dim at a time (~15GB peak) rather than a combined all-dims pass (OOM). - Update analysis/validation.ipynb to the path-based API; delete the analysis/export_*.py and compare_ode_steps_*.py one-off scripts. - Rewrite tests/test_analysis.py around parquet fixtures with an inline numpy oracle; add correlation/streaming-plotter and approx-median coverage. Verified end-to-end on the 32GB predict file: full notebook completes at ~25GB peak (no OOM); event rollup runs at ~13 cores. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

lbogner (Migrated from gitlab.etp.kit.edu) merged commit a6bb142a40 into master

2026-07-17 12:34:06 +02:00

lbogner commented

2026-07-17 12:34:07 +02:00

(Migrated from gitlab.etp.kit.edu)

mentioned in commit a6bb142a40

mentioned in commit a6bb142a4086c067003335d8a14d1c854702ec6a

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#12