Drop orphaned child tracks instead of nulling secondary targets
A listed child_track_id can fail to match any first-step row (e.g. a secondary absorbed below the tracking threshold at birth). The parent->child left join in _add_secondary_attributes left these as nulls, which silently became NaN once the parquet round-tripped through the loader's float32 padding — poisoning every later secondary slot in that step via the cumulative "remaining budget" in encode_secondaries, while e_sec quietly undercounted and n_sec (from len(child_track_ids)) overcounted relative to the actual lists. Drop orphans from both the per-secondary lists and child_track_ids itself so downstream counts stay consistent, and thread the per-file orphaned count back through convert_steps_to_parquet so both the sequential and --jobs>1 batch paths in `dwarf convert` can report an aggregate total instead of relying on grepping printed output. Also floors encode_secondaries' slot-0 budget to _EPS (matching the i>0 branch), fixing a harmless but noisy 0/0 divide warning on zero-secondary steps. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -26,6 +26,12 @@ from pathlib import Path
|
||||
GEN_RE = re.compile(r"^gen\d+$")
|
||||
SCHEMA_RE = re.compile(r"^schema(\d+)$")
|
||||
|
||||
# Matches the per-file orphan-drop message printed by
|
||||
# steps_to_parquet._add_secondary_attributes — each subprocess's count is
|
||||
# parsed back out of its captured stdout since there's no in-process return
|
||||
# value across the subprocess boundary.
|
||||
_ORPHAN_RE = re.compile(r"dropping (\d+) orphaned child_track_id")
|
||||
|
||||
|
||||
class DestinationError(ValueError):
|
||||
pass
|
||||
@@ -210,4 +216,9 @@ def run_parallel_job(
|
||||
print(f" {root_file}", file=sys.stderr)
|
||||
raise SystemExit(1)
|
||||
|
||||
total_orphaned = sum(
|
||||
int(m.group(1)) for _, _, stdout, _ in results for m in _ORPHAN_RE.finditer(stdout)
|
||||
)
|
||||
if total_orphaned:
|
||||
print(f"\n{total_orphaned} orphaned child track(s) dropped across {len(results)} file(s).")
|
||||
print(f"\nAll {len(results)} conversion(s) completed.")
|
||||
|
||||
Reference in New Issue
Block a user