analyze: normalize pdg dtype in open_side to fix rollout/reference concat
CI / Lint (ruff check) (push) Successful in 58s
CI / Format (ruff format) (push) Successful in 1m3s
CI / Type check (ty) (push) Successful in 1m4s
CI / Tests (push) Successful in 1m48s
CI / Lint (ruff check) (pull_request) Successful in 1m2s
CI / Format (ruff format) (pull_request) Successful in 1m6s
CI / Type check (ty) (pull_request) Successful in 1m18s
CI / Tests (pull_request) Successful in 1m50s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped
CI / Lint (ruff check) (push) Successful in 58s
CI / Format (ruff format) (push) Successful in 1m3s
CI / Type check (ty) (push) Successful in 1m4s
CI / Tests (push) Successful in 1m48s
CI / Lint (ruff check) (pull_request) Successful in 1m2s
CI / Format (ruff format) (pull_request) Successful in 1m6s
CI / Type check (ty) (pull_request) Successful in 1m18s
CI / Tests (pull_request) Successful in 1m50s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped
Rollout output and the reference file's ROOT-derived parquet disagree on pdg's integer width (Int32 vs Int64), which only surfaced downstream as a pl.concat SchemaError in build_context's pdg-count merge. Cast to a canonical Int64 at the single scan entry point instead.
This commit is contained in:
@@ -106,18 +106,27 @@ def open_side(source: str | Path | pl.LazyFrame, side: Side) -> pl.LazyFrame:
|
|||||||
can push their own narrow projection into the parquet read — the single
|
can push their own narrow projection into the parquet read — the single
|
||||||
biggest lever on a larger-than-RAM file. ``pl.LazyFrame`` inputs pass straight
|
biggest lever on a larger-than-RAM file. ``pl.LazyFrame`` inputs pass straight
|
||||||
through (used by tests).
|
through (used by tests).
|
||||||
|
|
||||||
|
``pdg`` is cast to a canonical ``Int64`` here: the rollout writer and the
|
||||||
|
reference file's upstream ROOT→parquet conversion don't agree on integer
|
||||||
|
width, and an uncast mismatch only surfaces later as a ``pl.concat``
|
||||||
|
``SchemaError`` (e.g. in ``build_context``'s pdg-count merge).
|
||||||
"""
|
"""
|
||||||
if isinstance(source, pl.LazyFrame):
|
if isinstance(source, pl.LazyFrame):
|
||||||
return source
|
return source.with_columns(pl.col("pdg").cast(pl.Int64))
|
||||||
path = Path(source)
|
path = Path(source)
|
||||||
if side is Side.rollout:
|
if side is Side.rollout:
|
||||||
_check_rollout_metadata(path)
|
_check_rollout_metadata(path)
|
||||||
return pl.scan_parquet(path)
|
lf = pl.scan_parquet(path)
|
||||||
# The reference (a rollout's seed `dataset`) may be a directory of parquet
|
else:
|
||||||
# shards rather than a single file — scan them all.
|
# The reference (a rollout's seed `dataset`) may be a directory of
|
||||||
if path.is_dir():
|
# parquet shards rather than a single file — scan them all.
|
||||||
return pl.scan_parquet(str(path / "**/*.parquet"))
|
lf = (
|
||||||
return pl.scan_parquet(path)
|
pl.scan_parquet(str(path / "**/*.parquet"))
|
||||||
|
if path.is_dir()
|
||||||
|
else pl.scan_parquet(path)
|
||||||
|
)
|
||||||
|
return lf.with_columns(pl.col("pdg").cast(pl.Int64))
|
||||||
|
|
||||||
|
|
||||||
def physical_steps(lf: pl.LazyFrame, side: Side) -> pl.LazyFrame:
|
def physical_steps(lf: pl.LazyFrame, side: Side) -> pl.LazyFrame:
|
||||||
|
|||||||
Reference in New Issue
Block a user