event_id collisions across multi-file/multi-shard reference reads in giant analyze #22
Assignees
lars (Lars Bogner)
Clear assignees
lars (Lars Bogner)
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: lars/giant#22
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
giant/analysis/sources.py:open_side scans the reference side with pl.scan_parquet(str(path / "**/*.parquet")) when given a directory of shards, and never offsets or dedupes event_id across those shards. Since each source parquet file's event_id numbering restarts from 0 (one Geant4 job per file), events from different shards can collide on the same event_id, corrupting every downstream per-event grouping (group_by("event_id") in reduce.py/grouping.py/context.py, and the event_id % n_chunks condor chunking in catalog.py/condor.py).
This is the same root-cause bug as the training-pipeline event_id collision (fixed via a per-file index-based offset in giant/data/loader.py, giant/data/dataset.py, giant/data/setup_cache.py, giant/pipeline.py, and giant/cli.py's predict/_seed_from_data), but open_side's polars-based lazy multi-shard scan is a separate code path that fix didn't touch — it needs its own fix, likely using polars' include_file_paths on scan_parquet to compute a per-shard offset expression and add it to event_id before any grouping.