Add lazy polars I/O and duplicate KL/constraint checks for giant.analysis

lars/giant

load_predicted_local now reads predict parquet via a lazy polars scan with
column projection pushed into the reader, instead of materializing the
whole file as a pandas DataFrame. Also adds marginal_table_pl and
constraint_report_pl, polars-native duplicates that read straight from a
predict parquet path/LazyFrame and stay lazy per (group, dim) pair, so
peak memory is one column slice rather than the whole SampleCollection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

This commit is contained in:

Lars Bogner

2026-06-18 17:06:58 +02:00

parent a867fc4aae

commit 80b198c1a5

4 changed files with 230 additions and 13 deletions

									
										pyproject.toml
									
		+1
		
												View File
												
				@@ -23,6 +23,7 @@ convert = [

				]

				analysis = [

				    "matplotlib>=3.8",

				    "polars>=1.0",

				]

				[project.scripts]