fix(analysis): keep pre_E alive through prediction range subsampling #102

2026-09-09T14:24:28+02:00

lars commented

2026-09-09 14:24:28 +02:00

_row_subsample hash-filters on pre_E, but build_context's
prediction branch called it on a frame already projected down to
pred_<var>/true_<var> columns, so any run whose prediction file
exceeds sample_rows (the default is 1M; real predict outputs can be
100M+ rows) failed with ColumnNotFoundError: pre_E.

Subsample the full paired frame first, then project — matching every
other _row_subsample call site — and restructure the loop to subsample
once per prediction side instead of once per paired variable, cutting
6 streaming passes over the prediction file down to 1.

Add a regression test with sample_rows below the fixture's row count
so the hash-filter branch is actually exercised (the existing test
never took it).

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01Qb7xBAa6aAR94AzimgpPxq

`_row_subsample` hash-filters on `pre_E`, but `build_context`'s prediction branch called it on a frame already projected down to `pred_<var>`/`true_<var>` columns, so any run whose prediction file exceeds `sample_rows` (the default is 1M; real predict outputs can be 100M+ rows) failed with `ColumnNotFoundError: pre_E`. Subsample the full paired frame first, then project — matching every other _row_subsample call site — and restructure the loop to subsample once per prediction side instead of once per paired variable, cutting 6 streaming passes over the prediction file down to 1. Add a regression test with sample_rows below the fixture's row count so the hash-filter branch is actually exercised (the existing test never took it). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qb7xBAa6aAR94AzimgpPxq

lars added 1 commit 2026-09-09 14:24:29 +02:00

fix(analysis): keep pre_E alive through prediction range subsampling

CI / Sync project version with tag (hand-pushed tags only) (pull_request) Skipped

Details

CI / Publish package to Gitea package registry (pull_request) Skipped

Details

CI / Format (ruff format) (pull_request) Successful in 2m23s

Details

CI / Lint (ruff check) (pull_request) Successful in 2m33s

Details

CI / Type check (ty) (pull_request) Successful in 2m51s

Details

CI / Tests (pull_request) Successful in 6m35s

Details

CI / Release (bump, changelog, badges, tag) on merge to master (pull_request) Skipped

Details

c0cbc99231

`_row_subsample` hash-filters on `pre_E`, but `build_context`'s
prediction branch called it on a frame already projected down to
`pred_<var>`/`true_<var>` columns, so any run whose prediction file
exceeds `sample_rows` (the default is 1M; real predict outputs can be
100M+ rows) failed with `ColumnNotFoundError: pre_E`.

Subsample the full paired frame first, then project — matching every
other _row_subsample call site — and restructure the loop to subsample
once per prediction side instead of once per paired variable, cutting
6 streaming passes over the prediction file down to 1.

Add a regression test with sample_rows below the fixture's row count
so the hash-filter branch is actually exercised (the existing test
never took it).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qb7xBAa6aAR94AzimgpPxq

lars merged commit 8ac3060814 into master

2026-09-09 14:36:23 +02:00

lars deleted branch fix/analyze-prediction-subsample-pre-e

2026-09-09 14:36:23 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#102