Replace giant/analysis.py's dual numpy-SampleCollection + polars paths with a
single polars-streaming implementation that produces the validation notebook's
plots directly from a `giant predict --coord local` parquet, sized for files
larger than RAM.
- Drop the numpy SampleCollection path (load_predicted_local, marginal_table,
correlation_matrices, direction_alignment, constraint_report, plot_kl_bars)
and the rollout observables; the 5 remaining plotters now take a parquet
path / LazyFrame and stream internally.
- Rewrite compute_event_observables_pl to aggregate in parallel streaming
polars (post-pos reconstruction as expressions) instead of a serial
pyarrow-batch + numpy loop, fixing a pre-existing OOM (holistic median +
323M-row join in the bin-edge sizing). Medians are approximated from a
streaming log-bin histogram with within-bin interpolation.
- Keep every full-file scan narrow (few columns): on a file larger than RAM,
peak mmap memory, not scan count, is the binding constraint. Marginals run
one dim at a time (~15GB peak) rather than a combined all-dims pass (OOM).
- Update analysis/validation.ipynb to the path-based API; delete the
analysis/export_*.py and compare_ode_steps_*.py one-off scripts.
- Rewrite tests/test_analysis.py around parquet fixtures with an inline numpy
oracle; add correlation/streaming-plotter and approx-median coverage.
Verified end-to-end on the 32GB predict file: full notebook completes at
~25GB peak (no OOM); event rollup runs at ~13 cores.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add analysis scripts for the 10-vs-20 flow-matching ODE-step ablation on the
energy-conservation PoC predict outputs:
- compare_ode_steps_energy_conservation.py: per-event energy-budget table +
20-step plots and the 10-vs-20 overlay.
- compare_ode_steps_kl.py: per-step marginal KL(real||gen) per target dim over
fixed shared bins, so the two runs are directly comparable dim-by-dim.
Also commit export_energy_conservation_poc.py (the baseline event-level budget
export) and repoint validation.ipynb at the PoC predict file at sample_frac=1.0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Removes unused imports and an ambiguous variable name, narrows
Optional types before use so ty's flow analysis is satisfied, swaps
sum() over polars expressions for pl.sum_horizontal to avoid the
Literal[0] fallback type, and converts numpy bin edges to plain lists
before passing to matplotlib's hist (whose stub only accepts
Sequence[float]). Also applies ruff format across the repo, which had
drifted out of sync with the formatter.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Move ipykernel into the analysis extra instead of a separate
dependency group, since it's needed wherever analysis plotting runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pdg_contribution_table_pl sums real/generated total deposited energy and
total step_length per pdg species over the whole file (pure lazy polars
group_by, no post_pos reconstruction needed for these scalars). Adds
plot_pdg_energy_share/plot_pdg_length_share, each rendering two pies
(real vs generated) so the per-species breakdown can be compared directly,
plus a matching notebook section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sum(step_length) per event_id, alongside the existing total deposited
energy, since path length and energy deposit aren't interchangeable once
tracks scatter. Adds plot_total_length and a matching notebook cell.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds section headers and explanatory comments throughout the existing
tiers, plus new cells running compute_event_observables_pl and the four
event-level shower plots. Notebook outputs reflect the user's own re-run.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds plot_kl_bars/plot_kl_bars_pl (numpy/polars variants) for ranking which
target dimension or pdg/material stratum drives KL regressions, with the
same kl*n group capping as plot_marginals. Switches existing histogram
plots to step-type/log-scale. Adds sample_frac to load_predicted_local for
subsampling large predict parquets.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lars