train/rollout: submit as remote-GPU HTCondor jobs on TOpAS/NEMO2
CI / Lint (ruff check) (push) Successful in 1m2s
CI / Format (ruff format) (push) Failing after 1m11s
CI / Type check (ty) (push) Successful in 1m6s
CI / Tests (push) Successful in 1m57s
CI / Bump version, build & publish wheel (push) Has been skipped

Adds `giant train-submit`/`giant rollout-submit`, mirroring `giant analyze
submit`'s CPU-job pattern but for single remote-GPU jobs: +RemoteJob/
RequestGPUs, TARGET.ProvidesEtpCeph instead of the local-only
ProvidesETPResources, and a self-contained condor/ run dir (wrapper, submit
description, and a CondorJobMeta sidecar recording what was submitted and
the assigned cluster id) so a run stays traceable after the fact. Training
jobs re-check for last.pt on every wrapper invocation so a preempted job
resumes instead of restarting.

Also adds `giant new-run` to scaffold a run's config.toml + run dir (with
collision-free naming via the new shared `default_out_dir`) ahead of
submission, and factors router-flag parsing into `_router_cli_overrides`
so `train` and `new-run` resolve it identically.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-24 13:12:01 +02:00
parent 1115301eb9
commit 430917d8f2
7 changed files with 1046 additions and 26 deletions
+11
View File
@@ -58,6 +58,17 @@ def test_each_call_produces_a_distinct_uuid(tmp_path):
assert uuid1 != uuid2
def test_pinned_pred_uuid_is_used_as_is(tmp_path):
# `rollout-submit` pins the uuid at submit time so its condor run-dir,
# the output filename, and the eventual YAML sidecar all agree.
data = tmp_path / "data.parquet"
pinned = "abcd1234-abcd-4abc-9abc-abcdabcdabcd"
out, _, pred_uuid = _resolve_prediction_output(data, None, pred_uuid=pinned)
assert pred_uuid == pinned
assert out.name == f"{pinned}.parquet"
def test_dataset_path_is_resolved(tmp_path):
data = tmp_path / "data.parquet"
_, dataset_path, _ = _resolve_prediction_output(data, None)