Offset event_id per file to avoid cross-file collisions
CI / Format (ruff format) (push) Successful in 25s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 23s
CI / Lint (ruff check) (pull_request) Successful in 27s
CI / Tests (push) Successful in 1m14s
CI / Format (ruff format) (pull_request) Successful in 27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (pull_request) Successful in 59s
CI / Format (ruff format) (push) Successful in 25s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 23s
CI / Lint (ruff check) (pull_request) Successful in 27s
CI / Tests (push) Successful in 1m14s
CI / Format (ruff format) (pull_request) Successful in 27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (pull_request) Successful in 59s
Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -22,13 +22,16 @@ import numpy as np
|
||||
|
||||
from giant import config
|
||||
from giant.constants import COND_DIM, K_MAX, PARTICLE_PHYS_DIM, SEC_SLOT_DIM, X_DIM
|
||||
from giant.data.loader import load_event_ids
|
||||
from giant.data.loader import event_id_offset, load_event_ids
|
||||
from giant.data.transforms import Normalizer, sorted_membership
|
||||
|
||||
# Bump manually on a change to the data-encoding semantics (e.g. a future
|
||||
# energy_simplex_encode bugfix) that doesn't also move one of _DIMS below —
|
||||
# a dims change already hard-invalidates on its own.
|
||||
_CACHE_FORMAT_VERSION = 1
|
||||
# v2: event_id is now offset per-file (see loader.event_id_offset) to avoid
|
||||
# cross-file collisions, so a v1 sidecar's event_index/normalizers were
|
||||
# computed against collided ids and must not be reused.
|
||||
_CACHE_FORMAT_VERSION = 2
|
||||
|
||||
_DIMS = {
|
||||
"COND_DIM": COND_DIM,
|
||||
@@ -263,7 +266,9 @@ def compute_event_index_from_files(files: list[Path]) -> tuple[np.ndarray, np.nd
|
||||
"""Unique event ids + per-event row (step) counts, across all `files`."""
|
||||
if not files:
|
||||
return np.empty(0, dtype=np.int64), np.empty(0, dtype=np.int64)
|
||||
all_ids = np.concatenate([load_event_ids(f) for f in files])
|
||||
all_ids = np.concatenate(
|
||||
[load_event_ids(f, offset=event_id_offset(i)) for i, f in enumerate(files)]
|
||||
)
|
||||
unique_ids, counts = np.unique(all_ids, return_counts=True)
|
||||
return unique_ids, counts
|
||||
|
||||
|
||||
Reference in New Issue
Block a user