Fix CLI/tooling robustness gaps and dedupe the Conditioning enum
CI / Lint (ruff check) (push) Successful in 35s
CI / Format (ruff format) (push) Failing after 31s
CI / Type check (ty) (push) Successful in 30s
CI / Sync project version with tag (push) Has been skipped
CI / Tests (push) Successful in 1m7s

- run_create_manifest gains --force; it previously overwrote an
  existing manifest (including holdout.manifest, which
  check_holdout_overlap exists specifically to protect) with no
  warning or backup on a second run.
- _git_user_name only caught OSError, not subprocess.TimeoutExpired (a
  SubprocessError, not an OSError) — a slow/loaded shared portal
  machine could crash `dwarf bump-gen`/`bump-schema` instead of
  degrading to by=None as intended.
- `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when
  the requested count exceeds ~1/4 of the machine's CPUs, matching the
  same shared-machine etiquette check added to giant train in the
  previous commit.
- The Conditioning enum was independently redefined in both
  giant/cli.py and scripts/dwarf.py; moved to a single
  giant.config.Conditioning both now import, removing the drift risk
  of a third conditioning mode being added to one but not the other.

Each fix has a regression test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-03 13:49:09 +02:00
parent ad1b8e7835
commit ca3a2a3462
6 changed files with 124 additions and 10 deletions
+17 -2
View File
@@ -85,7 +85,12 @@ def _git_user_name() -> str | None:
out = subprocess.run(
["git", "config", "user.name"], capture_output=True, text=True, timeout=2
)
except OSError:
except (OSError, subprocess.SubprocessError):
# OSError (e.g. git not on PATH) and subprocess.SubprocessError
# (e.g. TimeoutExpired) are unrelated hierarchies — TimeoutExpired
# is not an OSError, so catching only OSError (as before) let a
# slow/loaded NFS-backed portal machine crash this instead of
# degrading to by=None as intended.
return None
name = out.stdout.strip()
return name or None
@@ -730,6 +735,7 @@ def run_create_manifest(
pool: str | None = None,
type_: str | None = None,
root: str = "/ceph/lbogner/geant_steps",
force: bool = False,
) -> None:
if (output is None) == (pool is None):
raise SystemExit("error: exactly one of --output or --pool is required")
@@ -745,6 +751,12 @@ def run_create_manifest(
parquet_files = [Path(f) for f in files]
lines, missing, resolved = plan_create_manifest(output_path, parquet_files)
overlaps = check_holdout_overlap(output_path, resolved)
# Unlike missing/overlaps this is a hard stop even without --execute
# reaching the write, since create_manifest has no in-place "update" mode
# (unlike update_manifest) — a second run against the same output_path
# (e.g. holdout.manifest, the file check_holdout_overlap exists to
# protect) would otherwise silently clobber it with no diff/backup.
already_exists = output_path.exists() and not force
print(f"=== {'EXECUTING' if execute else 'DRY RUN'} ===")
print(f"manifest: {output_path.resolve()}")
@@ -761,7 +773,10 @@ def run_create_manifest(
for name, f in overlaps:
print(f" {f} (also in {name})")
if (missing or overlaps) and execute:
if already_exists:
print(f"\n{output_path} already exists — pass --force to overwrite it.")
if (missing or overlaps or already_exists) and execute:
raise SystemExit("error: refusing to write manifest (see above)")
if not execute: