train/rollout: submit as remote-GPU HTCondor jobs on TOpAS/NEMO2
Adds `giant train-submit`/`giant rollout-submit`, mirroring `giant analyze submit`'s CPU-job pattern but for single remote-GPU jobs: +RemoteJob/ RequestGPUs, TARGET.ProvidesEtpCeph instead of the local-only ProvidesETPResources, and a self-contained condor/ run dir (wrapper, submit description, and a CondorJobMeta sidecar recording what was submitted and the assigned cluster id) so a run stays traceable after the fact. Training jobs re-check for last.pt on every wrapper invocation so a preempted job resumes instead of restarting. Also adds `giant new-run` to scaffold a run's config.toml + run dir (with collision-free naming via the new shared `default_out_dir`) ahead of submission, and factors router-flag parsing into `_router_cli_overrides` so `train` and `new-run` resolve it identically. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -58,6 +58,17 @@ def test_each_call_produces_a_distinct_uuid(tmp_path):
|
||||
assert uuid1 != uuid2
|
||||
|
||||
|
||||
def test_pinned_pred_uuid_is_used_as_is(tmp_path):
|
||||
# `rollout-submit` pins the uuid at submit time so its condor run-dir,
|
||||
# the output filename, and the eventual YAML sidecar all agree.
|
||||
data = tmp_path / "data.parquet"
|
||||
pinned = "abcd1234-abcd-4abc-9abc-abcdabcdabcd"
|
||||
out, _, pred_uuid = _resolve_prediction_output(data, None, pred_uuid=pinned)
|
||||
|
||||
assert pred_uuid == pinned
|
||||
assert out.name == f"{pinned}.parquet"
|
||||
|
||||
|
||||
def test_dataset_path_is_resolved(tmp_path):
|
||||
data = tmp_path / "data.parquet"
|
||||
_, dataset_path, _ = _resolve_prediction_output(data, None)
|
||||
|
||||
Reference in New Issue
Block a user