WIP: Condor gpu train rollout #17

Draft
lars wants to merge 4 commits from condor-gpu-train-rollout into master

4 Commits

Author SHA1 Message Date
lars 91aed19d4b Merge remote-tracking branch 'origin/master' into condor-gpu-train-rollout
CI / Lint (ruff check) (push) Successful in 26s
CI / Format (ruff format) (push) Successful in 26s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 21s
CI / Tests (push) Successful in 53s
CI / Lint (ruff check) (pull_request) Successful in 25s
CI / Format (ruff format) (pull_request) Successful in 25s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 21s
CI / Tests (pull_request) Successful in 53s
# Conflicts:
#	CLAUDE.md
2026-07-27 15:01:54 +02:00
lars dde8b367a4 style: apply ruff format to cli.py and test_condor_gpu.py
CI / Lint (ruff check) (push) Successful in 1m1s
CI / Format (ruff format) (push) Successful in 1m9s
CI / Type check (ty) (push) Successful in 1m11s
CI / Tests (push) Successful in 1m56s
CI / Bump version, build & publish wheel (push) Has been skipped
Pre-existing formatting drift, unrelated to any functional change.
2026-07-24 13:17:58 +02:00
lars 3d1fa7979e docs: bring README in line with current architecture and CLI
Adds the physical/embedding conditioning split, WGAN/MoE-routing modes,
HTCondor remote-GPU submission, and the giant analyze pipeline, none of
which were reflected in the previous version. Updates the project
structure listing to match the current module layout.
2026-07-24 13:16:47 +02:00
lars 430917d8f2 train/rollout: submit as remote-GPU HTCondor jobs on TOpAS/NEMO2
CI / Lint (ruff check) (push) Successful in 1m2s
CI / Format (ruff format) (push) Failing after 1m11s
CI / Type check (ty) (push) Successful in 1m6s
CI / Tests (push) Successful in 1m57s
CI / Bump version, build & publish wheel (push) Has been skipped
Adds `giant train-submit`/`giant rollout-submit`, mirroring `giant analyze
submit`'s CPU-job pattern but for single remote-GPU jobs: +RemoteJob/
RequestGPUs, TARGET.ProvidesEtpCeph instead of the local-only
ProvidesETPResources, and a self-contained condor/ run dir (wrapper, submit
description, and a CondorJobMeta sidecar recording what was submitted and
the assigned cluster id) so a run stays traceable after the fact. Training
jobs re-check for last.pt on every wrapper invocation so a preempted job
resumes instead of restarting.

Also adds `giant new-run` to scaffold a run's config.toml + run dir (with
collision-free naming via the new shared `default_out_dir`) ahead of
submission, and factors router-flag parsing into `_router_cli_overrides`
so `train` and `new-run` resolve it identically.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-24 13:12:01 +02:00