Add data-integrity guards against silent NaN/Inf propagation and races
- log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -13,6 +13,7 @@ before reuse — see `load`/`save`.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import json
|
||||
import os
|
||||
from dataclasses import dataclass, field
|
||||
@@ -270,15 +271,32 @@ def save(
|
||||
Best-effort: any OSError (permission denied on a read-only mount, disk
|
||||
full, ...) is caught, echoed as a warning, and swallowed — a failure to
|
||||
cache must never fail training.
|
||||
|
||||
The load-merge-write is serialized with an exclusive flock on a sidecar
|
||||
lockfile: `os.replace` alone only guarantees the *file* is never
|
||||
corrupt, not that concurrent writers don't race. Without the lock, two
|
||||
concurrent `giant train`/condor jobs against the same `data` path (this
|
||||
repo's shared-portal/condor usage makes that a real scenario, not just
|
||||
theoretical) could both `load()` the same base state, merge their own
|
||||
`sections` in independently, and whichever `os.replace()` lands last
|
||||
silently discards the other's freshly-computed section.
|
||||
"""
|
||||
path = sidecar_path(data)
|
||||
lock_path = path.parent / f".{path.name}.lock"
|
||||
tmp = path.parent / f".{path.name}.tmp.{os.getpid()}"
|
||||
try:
|
||||
base = load(data, files, echo=lambda *a, **k: None) or SetupCache.empty(files)
|
||||
merged = base.merge(sections)
|
||||
payload = json.dumps(merged.to_json(), separators=(",", ":"))
|
||||
tmp.write_text(payload)
|
||||
os.replace(tmp, path)
|
||||
with open(lock_path, "a") as lock_file:
|
||||
fcntl.flock(lock_file, fcntl.LOCK_EX)
|
||||
try:
|
||||
base = load(data, files, echo=lambda *a, **k: None) or SetupCache.empty(
|
||||
files
|
||||
)
|
||||
merged = base.merge(sections)
|
||||
payload = json.dumps(merged.to_json(), separators=(",", ":"))
|
||||
tmp.write_text(payload)
|
||||
os.replace(tmp, path)
|
||||
finally:
|
||||
fcntl.flock(lock_file, fcntl.LOCK_UN)
|
||||
except OSError as exc:
|
||||
echo(
|
||||
f"setup cache: could not write {path} ({exc}) — continuing without caching"
|
||||
|
||||
Reference in New Issue
Block a user