WIP: Condor gpu train rollout #17

Draft

lars wants to merge 4 commits from condor-gpu-train-rollout into master

pull from: condor-gpu-train-rollout

merge into: lars:master

lars:master

lars:v0.3.0-stage2-autoregressive

lars:feat/router-learnable-width

lars:fix/event-id-file-offset

Conversation 0 Commits 4 Files Changed 8

+1105 -46

lars commented 2026-07-27 15:06:37 +02:00

Owner

Copy Link

Copy Source

No description provided.

lars added 4 commits 2026-07-27 15:06:37 +02:00

train/rollout: submit as remote-GPU HTCondor jobs on TOpAS/NEMO2 ... 430917d8f2

Adds `giant train-submit`/`giant rollout-submit`, mirroring `giant analyze
submit`'s CPU-job pattern but for single remote-GPU jobs: +RemoteJob/
RequestGPUs, TARGET.ProvidesEtpCeph instead of the local-only
ProvidesETPResources, and a self-contained condor/ run dir (wrapper, submit
description, and a CondorJobMeta sidecar recording what was submitted and
the assigned cluster id) so a run stays traceable after the fact. Training
jobs re-check for last.pt on every wrapper invocation so a preempted job
resumes instead of restarting.

Also adds `giant new-run` to scaffold a run's config.toml + run dir (with
collision-free naming via the new shared `default_out_dir`) ahead of
submission, and factors router-flag parsing into `_router_cli_overrides`
so `train` and `new-run` resolve it identically.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

docs: bring README in line with current architecture and CLI ... 3d1fa7979e

Adds the physical/embedding conditioning split, WGAN/MoE-routing modes,
HTCondor remote-GPU submission, and the giant analyze pipeline, none of
which were reflected in the previous version. Updates the project
structure listing to match the current module layout.

style: apply ruff format to cli.py and test_condor_gpu.py ... dde8b367a4

Pre-existing formatting drift, unrelated to any functional change.

Merge remote-tracking branch 'origin/master' into condor-gpu-train-rollout ... 91aed19d4b

# Conflicts:
#	CLAUDE.md

Some checks are pending

Hide all checks

CI / Lint (ruff check) (push) Successful in 26s

Details

CI / Format (ruff format) (push) Successful in 26s

Details

CI / Sync project version with tag (push) Has been skipped

Details

CI / Type check (ty) (push) Successful in 21s

Details

CI / Tests (push) Successful in 53s

Details

CI / Lint (ruff check) (pull_request) Successful in 25s

Details

CI / Format (ruff format) (pull_request) Successful in 25s

Details

CI / Sync project version with tag (pull_request) Has been skipped

Details

CI / Type check (ty) (pull_request) Successful in 21s

Details

CI / Tests (pull_request) Successful in 53s

Details

This pull request is marked as a work in progress.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin condor-gpu-train-rollout:condor-gpu-train-rollout

git checkout condor-gpu-train-rollout

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

experiment

investigation

performance

phase2

validation

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

lars (Lars Bogner)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#17