Add data-integrity guards against silent NaN/Inf propagation and races
- log_transform / _validate_unit_pre_dir now raise on non-finite input instead of letting a NaN row silently poison the persisted normalizer cache (norm < 1e-6 was always False for NaN, so the existing guard never caught it). - encode_secondaries warns when a row's secondary energies cumulatively exceed e_sec, instead of silently saturating the overflowing slot's stick-breaking logit via the _EPS floor. - EVENT_ID_FILE_STRIDE overflow now raises instead of silently colliding two files' event ids together (reintroducing train/val leakage). - make_event_split(val_fraction=0.0) now actually holds out nothing, instead of always forcing at least 1 validation event. - setup_cache.save() is now serialized with a flock, since two concurrent writers (a real scenario on this repo's shared portal/condor machines) could otherwise race and silently drop one writer's freshly-computed cache section. - Documented (no behavior change) the pre_dir ≈ -ẑ antipodal rotation singularity in _rodrigues_axis, which is real but inherent to any single-valued local-frame convention. Each fix has a regression test. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -214,6 +214,41 @@ def test_save_merges_non_colliding_normalizer_keys(tmp_path):
|
||||
assert loaded.normalizers["k2"].n_train_steps == 2
|
||||
|
||||
|
||||
def test_save_is_serialized_against_concurrent_writers(tmp_path):
|
||||
"""Without the flock in setup_cache.save(), two concurrent writers can
|
||||
both load() the same base state and merge their own section in
|
||||
independently, so whichever os.replace() lands last silently drops the
|
||||
other's key — a lost-update race, not a corrupt file. Each of these
|
||||
threads writes a distinct normalizer key many times over; if the
|
||||
load-merge-write critical section isn't actually serialized, at least
|
||||
one thread's key is likely to go missing from the final merged cache."""
|
||||
import threading
|
||||
|
||||
data = _touch_parquet(tmp_path / "shard.parquet")
|
||||
files = [data]
|
||||
setup_cache.save(data, files, SetupCache.empty(files))
|
||||
|
||||
n_writers, n_rounds = 6, 15
|
||||
|
||||
def _writer(idx: int) -> None:
|
||||
for r in range(n_rounds):
|
||||
cache = SetupCache.empty(files)
|
||||
cache.normalizers[f"k{idx}"] = _entry(n_train_steps=r)
|
||||
setup_cache.save(data, files, cache)
|
||||
|
||||
threads = [threading.Thread(target=_writer, args=(i,)) for i in range(n_writers)]
|
||||
for t in threads:
|
||||
t.start()
|
||||
for t in threads:
|
||||
t.join()
|
||||
|
||||
loaded = setup_cache.load(data, files)
|
||||
assert loaded is not None
|
||||
assert set(loaded.normalizers.keys()) == {f"k{i}" for i in range(n_writers)}
|
||||
for i in range(n_writers):
|
||||
assert loaded.normalizers[f"k{i}"].n_train_steps == n_rounds - 1
|
||||
|
||||
|
||||
# ── energy_quantiles_from_sample / energy_quantile_at ───────────────────
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user