Add data-integrity guards against silent NaN/Inf propagation and races
CI / Lint (ruff check) (push) Successful in 33s
CI / Format (ruff format) (push) Successful in 34s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 32s
CI / Tests (push) Successful in 2m12s

- log_transform / _validate_unit_pre_dir now raise on non-finite input
  instead of letting a NaN row silently poison the persisted normalizer
  cache (norm < 1e-6 was always False for NaN, so the existing guard
  never caught it).
- encode_secondaries warns when a row's secondary energies cumulatively
  exceed e_sec, instead of silently saturating the overflowing slot's
  stick-breaking logit via the _EPS floor.
- EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding
  two files' event ids together (reintroducing train/val leakage).
- make_event_split(val_fraction=0.0) now actually holds out nothing,
  instead of always forcing at least 1 validation event.
- setup_cache.save() is now serialized with a flock, since two
  concurrent writers (a real scenario on this repo's shared
  portal/condor machines) could otherwise race and silently drop one
  writer's freshly-computed cache section.
- Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation
  singularity in _rodrigues_axis, which is real but inherent to any
  single-valued local-frame convention.

Each fix has a regression test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-03 13:47:29 +02:00
parent a4c0443e01
commit 74343d3e48
8 changed files with 247 additions and 15 deletions
+23 -5
View File
@@ -13,6 +13,7 @@ before reuse — see `load`/`save`.
from __future__ import annotations
import fcntl
import json
import os
from dataclasses import dataclass, field
@@ -270,15 +271,32 @@ def save(
Best-effort: any OSError (permission denied on a read-only mount, disk
full, ...) is caught, echoed as a warning, and swallowed — a failure to
cache must never fail training.
The load-merge-write is serialized with an exclusive flock on a sidecar
lockfile: `os.replace` alone only guarantees the *file* is never
corrupt, not that concurrent writers don't race. Without the lock, two
concurrent `giant train`/condor jobs against the same `data` path (this
repo's shared-portal/condor usage makes that a real scenario, not just
theoretical) could both `load()` the same base state, merge their own
`sections` in independently, and whichever `os.replace()` lands last
silently discards the other's freshly-computed section.
"""
path = sidecar_path(data)
lock_path = path.parent / f".{path.name}.lock"
tmp = path.parent / f".{path.name}.tmp.{os.getpid()}"
try:
base = load(data, files, echo=lambda *a, **k: None) or SetupCache.empty(files)
merged = base.merge(sections)
payload = json.dumps(merged.to_json(), separators=(",", ":"))
tmp.write_text(payload)
os.replace(tmp, path)
with open(lock_path, "a") as lock_file:
fcntl.flock(lock_file, fcntl.LOCK_EX)
try:
base = load(data, files, echo=lambda *a, **k: None) or SetupCache.empty(
files
)
merged = base.merge(sections)
payload = json.dumps(merged.to_json(), separators=(",", ":"))
tmp.write_text(payload)
os.replace(tmp, path)
finally:
fcntl.flock(lock_file, fcntl.LOCK_UN)
except OSError as exc:
echo(
f"setup cache: could not write {path} ({exc}) — continuing without caching"