Offset event_id per file to avoid cross-file collisions
CI / Format (ruff format) (push) Successful in 25s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 23s
CI / Lint (ruff check) (pull_request) Successful in 27s
CI / Tests (push) Successful in 1m14s
CI / Format (ruff format) (pull_request) Successful in 27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (pull_request) Successful in 59s
CI / Format (ruff format) (push) Successful in 25s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 23s
CI / Lint (ruff check) (pull_request) Successful in 27s
CI / Tests (push) Successful in 1m14s
CI / Format (ruff format) (pull_request) Successful in 27s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 31s
CI / Tests (pull_request) Successful in 59s
Each input parquet file is one Geant4 job (scripts/steps_to_parquet.py), and a job's event_id numbering always restarts from 0 — so loading multiple files together (a directory or .manifest) let same-numbered events from different files collapse into one during the event index scan and train/val split, corrupting both. Every per-file event_id now gets offset by file index * EVENT_ID_FILE_STRIDE (giant/data/loader.py), threaded through the setup-cache event index, the streaming dataset, and predict/rollout seeding. Bumps the setup-cache format version so stale sidecars computed pre-fix are invalidated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -225,3 +225,35 @@ def test_n_train_steps_for_split_matches_full_scan():
|
||||
result = setup_cache.n_train_steps_for_split(unique_ids, counts, train_events_arr)
|
||||
|
||||
assert result == 20 + 40 + 50
|
||||
|
||||
|
||||
# ── compute_event_index_from_files: cross-file event_id offsetting ─────────
|
||||
|
||||
|
||||
def test_compute_event_index_from_files_offsets_colliding_ids(tmp_path):
|
||||
"""Two files that each restart event_id from 0 (one Geant4 job per file)
|
||||
must not have their same-numbered events collapsed into one by
|
||||
np.unique — each file's ids get shifted by a distinct offset first (see
|
||||
giant.data.loader.event_id_offset)."""
|
||||
path_a = tmp_path / "a.parquet"
|
||||
path_b = tmp_path / "b.parquet"
|
||||
pd.DataFrame({"event_id": [0, 1, 2]}).to_parquet(path_a)
|
||||
pd.DataFrame({"event_id": [0, 1, 2]}).to_parquet(path_b)
|
||||
|
||||
unique_ids, counts = setup_cache.compute_event_index_from_files([path_a, path_b])
|
||||
|
||||
assert len(unique_ids) == 6
|
||||
assert int(counts.sum()) == 6
|
||||
assert np.all(counts == 1)
|
||||
|
||||
|
||||
def test_compute_event_index_from_files_single_file_unaffected(tmp_path):
|
||||
"""A single file's ids are offset by 0 (event_id_offset(0) == 0), so a
|
||||
single-file load's unique ids/counts are unchanged by the offsetting."""
|
||||
path = tmp_path / "a.parquet"
|
||||
pd.DataFrame({"event_id": [5, 5, 7]}).to_parquet(path)
|
||||
|
||||
unique_ids, counts = setup_cache.compute_event_index_from_files([path])
|
||||
|
||||
np.testing.assert_array_equal(unique_ids, [5, 7])
|
||||
np.testing.assert_array_equal(counts, [2, 1])
|
||||
|
||||
Reference in New Issue
Block a user