analyze: normalize pdg dtype in open_side to fix rollout/reference concat
CI / Lint (ruff check) (push) Successful in 58s
CI / Format (ruff format) (push) Successful in 1m3s
CI / Type check (ty) (push) Successful in 1m4s
CI / Tests (push) Successful in 1m48s
CI / Lint (ruff check) (pull_request) Successful in 1m2s
CI / Format (ruff format) (pull_request) Successful in 1m6s
CI / Type check (ty) (pull_request) Successful in 1m18s
CI / Tests (pull_request) Successful in 1m50s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped

Rollout output and the reference file's ROOT-derived parquet disagree on
pdg's integer width (Int32 vs Int64), which only surfaced downstream as a
pl.concat SchemaError in build_context's pdg-count merge. Cast to a
canonical Int64 at the single scan entry point instead.
This commit is contained in:
2026-07-24 14:32:06 +02:00
parent 80d544aa73
commit 70d0f04326
+16 -7
View File
@@ -106,18 +106,27 @@ def open_side(source: str | Path | pl.LazyFrame, side: Side) -> pl.LazyFrame:
can push their own narrow projection into the parquet read — the single can push their own narrow projection into the parquet read — the single
biggest lever on a larger-than-RAM file. ``pl.LazyFrame`` inputs pass straight biggest lever on a larger-than-RAM file. ``pl.LazyFrame`` inputs pass straight
through (used by tests). through (used by tests).
``pdg`` is cast to a canonical ``Int64`` here: the rollout writer and the
reference file's upstream ROOT→parquet conversion don't agree on integer
width, and an uncast mismatch only surfaces later as a ``pl.concat``
``SchemaError`` (e.g. in ``build_context``'s pdg-count merge).
""" """
if isinstance(source, pl.LazyFrame): if isinstance(source, pl.LazyFrame):
return source return source.with_columns(pl.col("pdg").cast(pl.Int64))
path = Path(source) path = Path(source)
if side is Side.rollout: if side is Side.rollout:
_check_rollout_metadata(path) _check_rollout_metadata(path)
return pl.scan_parquet(path) lf = pl.scan_parquet(path)
# The reference (a rollout's seed `dataset`) may be a directory of parquet else:
# shards rather than a single file — scan them all. # The reference (a rollout's seed `dataset`) may be a directory of
if path.is_dir(): # parquet shards rather than a single file — scan them all.
return pl.scan_parquet(str(path / "**/*.parquet")) lf = (
return pl.scan_parquet(path) pl.scan_parquet(str(path / "**/*.parquet"))
if path.is_dir()
else pl.scan_parquet(path)
)
return lf.with_columns(pl.col("pdg").cast(pl.Int64))
def physical_steps(lf: pl.LazyFrame, side: Side) -> pl.LazyFrame: def physical_steps(lf: pl.LazyFrame, side: Side) -> pl.LazyFrame: