lars/giant
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Orchestrate the full pipeline with b2luigi (cache-warm → train → rollout → analysis) #83

New Issue
Open
opened 2026-08-26 10:21:25 +02:00 by lars · 0 comments
lars commented 2026-08-26 10:21:25 +02:00
Owner
Copy Link
Copy Source

b2luigi orchestration for GIANT

Context

Every multi-step GIANT experiment is chained by hand today: dwarf warm-cache → dwarf build-geometry-oracle → giant train → giant rollout → giant analyze prep/submit → wait → giant analyze render. Nothing tracks what has already been produced, nothing waits for anything, and the HTCondor layer is a hand-rolled submit-file generator (giant/analysis/condor.py: write_submit, jobs.txt, analyze.sub) that is fire-and-forget — analyze submit returns immediately and the user has to poll condor_q themselves and remember to run render afterwards. A second, parallel copy of the same idea for GPU jobs is half-finished on the condor-gpu-train-rollout branch (giant/condor.py, train-submit/rollout-submit).

This replaces all of it with b2luigi: a luigi wrapper that already does dependency resolution, target-based idempotency ("done" = output file exists), HTCondor submission with status polling, per-task resource settings, logging, and a central scheduler UI. After this change b2luigi is the only sanctioned way to run a multi-step pipeline; the giant/dwarf CLIs are reduced to single-step primitives that the tasks invoke, and the bespoke condor code is deleted.

Design

New package giant/workflow/

module contents
spec.py workflow TOML → frozen dataclasses (WorkflowSpec, TrainSpec, RolloutSpec, AnalysisSpec, CondorSpec), name-uniqueness + cross-reference validation, and spec_hash() per sub-spec
htcondor.py builds the htcondor_settings dicts (docker universe, +RemoteJob, +RequestWalltime, GPU requirement expressions)
tasks.py the b2luigi Task classes below
run.py the script b2luigi re-executes on workers: argparse --spec, settings wiring, b2luigi.process(WorkflowTask(...), workers=N, batch=...)

giant workflow run <spec.toml> [--batch] [--workers N] [--mode dry-run|show-output|remove] [--scheduler-host/--scheduler-port] (new sub-app in giant/cli.py) is a thin exec of run.py so there is one documented entry point; python giant/workflow/run.py … stays valid because b2luigi's executable wrapper needs a real script path, not python -m.

Workflow spec (one file, the only place a pipeline is parameterised)

[workflow]
name             = "baseline-vs-router"
result_dir       = "/ceph/lbogner/workflows/baseline-vs-router"   # b2luigi result_dir
log_dir          = "/ceph/lbogner/workflows/baseline-vs-router/logs"

[condor]
accounting_group = "cms"
repo_dir         = "/work/lbogner/giant"      # == working_dir
env_script       = "/work/lbogner/giant/condor_env.sh"
docker_image_cpu = "cverstege/alma9-gridjob"
docker_image_gpu = "mschnepf/slc7-condocker"
remote           = true

[dataset]
steps     = "/ceph/lbogner/geant_steps/train/"     # training data
reference = "/ceph/lbogner/geant_steps/holdout/"   # rollout seeds + analysis truth

[geometry]
method = "slab"
subsample = 500_000

[[train]]
name = "baseline"
config = "configs/baseline.toml"
epochs = 200
epochs_per_job = 1               # one condor job per epoch
request_gpus = 1
gpu_memory_mb = 20000
overrides = { lr = 3e-4 }        # passed through as `giant train` flags

[[rollout]]
name = "baseline"
train = "baseline"               # -> [[train]].name
n_events = 2000
energy_cutoff = 0.1

[[analysis]]
name = "baseline-vs-router"
rollouts = ["baseline", "router-balanced"]
chunks = 32
energy_bins = 4
bins = 50
top_pdg = 6
gallery = true

Every task carries two luigi parameters: its name and a short spec_hash (8 hex of the canonicalised resolved sub-spec, including transitive parents). The hash is what makes an edited spec produce a fresh result directory instead of silently reusing stale outputs. Set use_parameter_name_in_output = True so paths read …/name=baseline/spec_hash=1a2b3c4d/best.pt.

Task graph

DatasetTask (ExternalTask)  ──┬─> WarmCacheTask ──> TrainEpochTask(name, 1) -> … -> TrainEpochTask(name, N) -> TrainTask(name) ──┐
                              └─> GeometryOracleTask ──┐                                                                        │
                                                       └──> RolloutTask(name) ──┐
                                                                                v
                              AnalysisPrepTask(name) ──> AnalysisComputeTask(name, plot_id, chunk) ──> AnalysisRenderTask(name)
                                                                                                            ^
WorkflowTask (WrapperTask) ─────────────────────────────────────────────────────────────────────────────────┘
  • DatasetTask — b2luigi.ExternalTask over the steps parquet path; fails fast with a clear message if /ceph is not mounted.
  • WarmCacheTask — calls giant.tools.warm_setup_cache.run_warm_setup_cache in-process. Its real product (<data>.giant_train_cache.json) lives next to the dataset, not under result_dir, so the b2luigi target is a small stamp JSON recording the sidecar path + its mtime/size; CPU condor job (high memory, no GPU).
  • GeometryOracleTask — calls the dwarf build-geometry-oracle implementation (giant/tools/geometry_oracle.py); output oracle.pkl under result_dir. CPU job.
  • TrainEpochTask(name, epoch) — one short GPU condor job per epoch, chained: epoch k requires epoch k−1 (epoch 1 requires WarmCacheTask). Each job calls run_train_job with out_dir = its own output dir, epochs = k, and resume = <epoch k−1 dir>/last.pt. This needs no change to the training loop: giant/training/loop.py:160-164 already sets start_epoch = ckpt["epoch"] + 1 and returns cleanly if the checkpoint already covers --epochs, so --epochs k --resume <k−1>/last.pt runs exactly epoch k. --out already wins over resume.parent (giant/cli.py:716-732), so the per-epoch output dirs work as-is.
    • Target: last.pt. best.pt is written by the loop only when that epoch improved, and best_val_loss travels inside the checkpoint, so the global best comparison stays correct across jobs — "best.pt exists in epoch dir k" means exactly "epoch k was the best so far".
    • Why this is worth the plumbing: a ~200-epoch run becomes 200 jobs of tens of minutes instead of one multi-day job, which schedules far better on a busy farm, survives preemption without losing more than one epoch, and gives luigi a real per-epoch progress signal. The costs are one setup-stage + dataset scan per job (cheap because WarmCacheTask guarantees a cache hit — pass --cache-setup) and one queue wait per epoch; epochs_per_job in the spec (default 1) trades those back if the queue turns out to dominate.
    • To verify during implementation: whether the shuffle-buffer ordering is derived from --seed alone, in which case every epoch job would replay the same batch order. If so, derive the loader's shuffle seed from seed + epoch while leaving the val-split seed untouched (the split must stay identical across jobs).
  • TrainTask(name) — cheap local task requiring the final TrainEpochTask. Picks the highest-numbered epoch dir containing a best.pt, and publishes the run's canonical outputs into one directory: best.pt, last.pt, config.toml, and a metrics.csv concatenated from the per-epoch ones. Everything downstream (RolloutTask, humans, analyze metrics) points here and never has to know about the epoch fan-out.
  • RolloutTask(name) — GPU condor job; requires TrainTask + GeometryOracleTask. Targets are rollout.parquet and rollout.yaml.
  • AnalysisPrepTask(name) — local (cheap, streaming); requires every RolloutTask it names. Calls giant.analysis.prep with run_dir = its own output dir; targets shared.json + run_meta.json.
  • AnalysisComputeTask(name, plot_id, chunk) — one CPU condor job per (plot, chunk), replacing jobs.txt/analyze.sub entirely. The job set is enumerable ahead of time from catalog_ids() × chunks, collapsing to one chunk for the five chunkable=False specs (same rule as today's _job_walltimes, giant/analysis/condor.py:487). output() is an explicit LocalTarget on <prep_dir>/reduced_partial/<id>__<chunk>.json rather than add_to_output, so compute-one's existing on-disk contract is untouched and merge_one keeps working. htcondor_settings is a property, evaluated at submit time — i.e. after prep has run — so it can still read run_meta.json and set +RequestWalltime from giant/analysis/runtime_estimate.py:estimate_runtime_s.
  • AnalysisRenderTask(name) — always local (the only step importing plotstyle/LaTeX). Runs merge_all then render_run, plus gallery generate when gallery = true. Target: <prep_dir>/plots/metadata.yaml.
  • WorkflowTask — b2luigi.WrapperTask requiring one AnalysisRenderTask per [[analysis]].

Settings wiring (run.py)

result_dir, log_dir, task_file_dir from the spec; batch_system = "htcondor"; working_dir = repo_dir; env_script; executable = [".venv/bin/python"]. /ceph is shared between submit host and workers, so no transfer_files — result and log dirs must be on /ceph. AnalysisRenderTask overrides batch_system = "local" as a class property, which b2luigi honours per task.

GPU requirement strings are ported from condor-gpu-train-rollout:giant/condor.py::_gpu_requirements (TARGET.ProvidesEtpCeph =?= True ANDed with GPUs_DeviceName / GPUs_GlobalMemoryMb pins) rather than rewritten.

Deletions and CLI reduction

  • giant/analysis/condor.py: delete SubmitConfig, _WRAPPER, _submit_description, _job_walltimes, _resolve_giant_executable, write_submit. Keep prep, derive_run_dir, RunMeta, load_rollout_yaml(s), compute_reduced/compute_one, merge_one/merge_all — that is the real logic — and rename the module to giant/analysis/run.py since nothing in it submits any more. Update the re-exports in giant/analysis/__init__.py and its module docstring.
  • giant/cli.py: delete the analyze submit command (cli.py:1680-1756). prep, compute-one, merge-one, list, render, metrics stay as primitives.
  • Do not port train-submit / rollout-submit from condor-gpu-train-rollout — the workflow supersedes them, and that branch's giant/condor.py is reduced to the requirement-string helpers moved into giant/workflow/htcondor.py. This is a decision that branch's eventual merge must respect; note it in CLAUDE.md.
  • tests/test_condor.py: drop the write_submit/submit-description cases, keep the prep/merge ones.

Required change to giant rollout

_write_prediction_ref (giant/cli.py:188) writes the sidecar to <checkpoint.parent>/<random-uuid>.yaml, which is not a deterministic target. Change it so that when --out is passed explicitly, the sidecar goes to out.with_suffix(".yaml"); the existing uuid-under-the-checkpoint behaviour is kept for the no---out case so ad-hoc runs and the /ceph predictions convention are unaffected. Apply the same rule to giant predict for consistency.

Dependency

Add b2luigi>=1.0,<2 under a new workflow optional-dependency extra in pyproject.toml (it pulls luigi + tenacity), and include giant[workflow] in the dev extra. Document uv sync --extra cpu --extra workflow in CLAUDE.md.

Files

  • new: giant/workflow/{__init__,spec,htcondor,tasks,run}.py, tests/test_workflow_spec.py, tests/test_workflow_tasks.py, an example configs/workflow_example.toml
  • modified: giant/cli.py (delete analyze submit, add workflow sub-app, sidecar path rule), giant/analysis/condor.py → giant/analysis/run.py, giant/analysis/__init__.py, tests/test_condor.py, pyproject.toml, CLAUDE.md, README.md

Verification

  1. uv sync --extra cpu --extra workflow --extra dev, then uv run pytest, uv run ruff check ., uv run ty check ..
  2. giant workflow run configs/workflow_example.toml --mode dry-run on a local machine — asserts the whole DAG resolves and prints the tasks that would run; exit code 1 means work pending, 0 means everything already done.
  3. --mode show-output on the same spec — eyeball that every target path is where the plan says it is (green = exists, red = missing).
  4. Local end-to-end smoke on a tiny parquet (a few hundred events, 3 epochs, 1 rollout, chunks = 1, batch_system = "local"): confirm three epoch dirs each with a last.pt, the published best.pt/concatenated metrics.csv, rollout.parquet/rollout.yaml, reduced_partial/*.json, and plots/metadata.yaml, and that a second invocation is a no-op.
  5. Per-epoch chaining is equivalent to a single run: train the same tiny config for 3 epochs in one giant train invocation with a fixed seed and diff its metrics.csv against the workflow's concatenated one — the per-epoch losses should match (modulo the shuffle-seed question above).
  6. Idempotency/restart: delete the epoch-3 output dir and re-run — only epoch 3 onwards should re-execute; separately delete one reduced_partial/<id>__<chunk>.json and confirm exactly that one compute task re-runs, then render.
  7. On a portal machine, giant workflow run <real spec> --batch --workers 20 against the real dataset, optionally with luigid running for the progress UI; verify with condor_q -batch <job_name> that the (plot, chunk) job count matches len(catalog_ids()) × chunks (minus the chunk collapse for the five non-chunkable specs), and that render only fires after the last compute job succeeds.
# b2luigi orchestration for GIANT ## Context Every multi-step GIANT experiment is chained by hand today: `dwarf warm-cache` → `dwarf build-geometry-oracle` → `giant train` → `giant rollout` → `giant analyze prep/submit` → wait → `giant analyze render`. Nothing tracks what has already been produced, nothing waits for anything, and the HTCondor layer is a hand-rolled submit-file generator (`giant/analysis/condor.py`: `write_submit`, `jobs.txt`, `analyze.sub`) that is fire-and-forget — `analyze submit` returns immediately and the user has to poll `condor_q` themselves and remember to run `render` afterwards. A second, parallel copy of the same idea for GPU jobs is half-finished on the `condor-gpu-train-rollout` branch (`giant/condor.py`, `train-submit`/`rollout-submit`). This replaces all of it with [b2luigi](https://github.com/belle2/b2luigi): a luigi wrapper that already does dependency resolution, target-based idempotency ("done" = output file exists), HTCondor submission with status polling, per-task resource settings, logging, and a central scheduler UI. After this change b2luigi is the only sanctioned way to run a multi-step pipeline; the `giant`/`dwarf` CLIs are reduced to single-step primitives that the tasks invoke, and the bespoke condor code is deleted. ## Design ### New package `giant/workflow/` | module | contents | |---|---| | `spec.py` | workflow TOML → frozen dataclasses (`WorkflowSpec`, `TrainSpec`, `RolloutSpec`, `AnalysisSpec`, `CondorSpec`), name-uniqueness + cross-reference validation, and `spec_hash()` per sub-spec | | `htcondor.py` | builds the `htcondor_settings` dicts (docker universe, `+RemoteJob`, `+RequestWalltime`, GPU requirement expressions) | | `tasks.py` | the b2luigi `Task` classes below | | `run.py` | the script b2luigi re-executes on workers: argparse `--spec`, settings wiring, `b2luigi.process(WorkflowTask(...), workers=N, batch=...)` | `giant workflow run <spec.toml> [--batch] [--workers N] [--mode dry-run|show-output|remove] [--scheduler-host/--scheduler-port]` (new sub-app in `giant/cli.py`) is a thin exec of `run.py` so there is one documented entry point; `python giant/workflow/run.py …` stays valid because b2luigi's executable wrapper needs a real script path, not `python -m`. ### Workflow spec (one file, the only place a pipeline is parameterised) ```toml [workflow] name = "baseline-vs-router" result_dir = "/ceph/lbogner/workflows/baseline-vs-router" # b2luigi result_dir log_dir = "/ceph/lbogner/workflows/baseline-vs-router/logs" [condor] accounting_group = "cms" repo_dir = "/work/lbogner/giant" # == working_dir env_script = "/work/lbogner/giant/condor_env.sh" docker_image_cpu = "cverstege/alma9-gridjob" docker_image_gpu = "mschnepf/slc7-condocker" remote = true [dataset] steps = "/ceph/lbogner/geant_steps/train/" # training data reference = "/ceph/lbogner/geant_steps/holdout/" # rollout seeds + analysis truth [geometry] method = "slab" subsample = 500_000 [[train]] name = "baseline" config = "configs/baseline.toml" epochs = 200 epochs_per_job = 1 # one condor job per epoch request_gpus = 1 gpu_memory_mb = 20000 overrides = { lr = 3e-4 } # passed through as `giant train` flags [[rollout]] name = "baseline" train = "baseline" # -> [[train]].name n_events = 2000 energy_cutoff = 0.1 [[analysis]] name = "baseline-vs-router" rollouts = ["baseline", "router-balanced"] chunks = 32 energy_bins = 4 bins = 50 top_pdg = 6 gallery = true ``` Every task carries two luigi parameters: its `name` and a short `spec_hash` (8 hex of the canonicalised resolved sub-spec, including transitive parents). The hash is what makes an edited spec produce a fresh result directory instead of silently reusing stale outputs. Set `use_parameter_name_in_output = True` so paths read `…/name=baseline/spec_hash=1a2b3c4d/best.pt`. ### Task graph ``` DatasetTask (ExternalTask) ──┬─> WarmCacheTask ──> TrainEpochTask(name, 1) -> … -> TrainEpochTask(name, N) -> TrainTask(name) ──┐ └─> GeometryOracleTask ──┐ │ └──> RolloutTask(name) ──┐ v AnalysisPrepTask(name) ──> AnalysisComputeTask(name, plot_id, chunk) ──> AnalysisRenderTask(name) ^ WorkflowTask (WrapperTask) ─────────────────────────────────────────────────────────────────────────────────┘ ``` - **`DatasetTask`** — `b2luigi.ExternalTask` over the steps parquet path; fails fast with a clear message if `/ceph` is not mounted. - **`WarmCacheTask`** — calls `giant.tools.warm_setup_cache.run_warm_setup_cache` in-process. Its real product (`<data>.giant_train_cache.json`) lives next to the dataset, not under `result_dir`, so the b2luigi target is a small stamp JSON recording the sidecar path + its mtime/size; CPU condor job (high memory, no GPU). - **`GeometryOracleTask`** — calls the `dwarf build-geometry-oracle` implementation (`giant/tools/geometry_oracle.py`); output `oracle.pkl` under `result_dir`. CPU job. - **`TrainEpochTask(name, epoch)`** — **one short GPU condor job per epoch**, chained: epoch *k* requires epoch *k−1* (epoch 1 requires `WarmCacheTask`). Each job calls `run_train_job` with `out_dir` = its own output dir, `epochs = k`, and `resume = <epoch k−1 dir>/last.pt`. This needs **no change to the training loop**: `giant/training/loop.py:160-164` already sets `start_epoch = ckpt["epoch"] + 1` and returns cleanly if the checkpoint already covers `--epochs`, so `--epochs k --resume <k−1>/last.pt` runs exactly epoch *k*. `--out` already wins over `resume.parent` (`giant/cli.py:716-732`), so the per-epoch output dirs work as-is. - Target: `last.pt`. `best.pt` is written by the loop *only when that epoch improved*, and `best_val_loss` travels inside the checkpoint, so the global best comparison stays correct across jobs — "`best.pt` exists in epoch dir *k*" means exactly "epoch *k* was the best so far". - Why this is worth the plumbing: a ~200-epoch run becomes 200 jobs of tens of minutes instead of one multi-day job, which schedules far better on a busy farm, survives preemption without losing more than one epoch, and gives luigi a real per-epoch progress signal. The costs are one setup-stage + dataset scan per job (cheap because `WarmCacheTask` guarantees a cache hit — pass `--cache-setup`) and one queue wait per epoch; `epochs_per_job` in the spec (default 1) trades those back if the queue turns out to dominate. - **To verify during implementation**: whether the shuffle-buffer ordering is derived from `--seed` alone, in which case every epoch job would replay the same batch order. If so, derive the loader's shuffle seed from `seed + epoch` while leaving the val-split seed untouched (the split must stay identical across jobs). - **`TrainTask(name)`** — cheap local task requiring the final `TrainEpochTask`. Picks the highest-numbered epoch dir containing a `best.pt`, and publishes the run's canonical outputs into one directory: `best.pt`, `last.pt`, `config.toml`, and a `metrics.csv` concatenated from the per-epoch ones. Everything downstream (`RolloutTask`, humans, `analyze metrics`) points here and never has to know about the epoch fan-out. - **`RolloutTask(name)`** — GPU condor job; requires `TrainTask` + `GeometryOracleTask`. Targets are `rollout.parquet` and `rollout.yaml`. - **`AnalysisPrepTask(name)`** — local (cheap, streaming); requires every `RolloutTask` it names. Calls `giant.analysis.prep` with `run_dir` = its own output dir; targets `shared.json` + `run_meta.json`. - **`AnalysisComputeTask(name, plot_id, chunk)`** — one CPU condor job per (plot, chunk), replacing `jobs.txt`/`analyze.sub` entirely. The job set is enumerable ahead of time from `catalog_ids()` × `chunks`, collapsing to one chunk for the five `chunkable=False` specs (same rule as today's `_job_walltimes`, `giant/analysis/condor.py:487`). `output()` is an explicit `LocalTarget` on `<prep_dir>/reduced_partial/<id>__<chunk>.json` rather than `add_to_output`, so `compute-one`'s existing on-disk contract is untouched and `merge_one` keeps working. `htcondor_settings` is a *property*, evaluated at submit time — i.e. after prep has run — so it can still read `run_meta.json` and set `+RequestWalltime` from `giant/analysis/runtime_estimate.py:estimate_runtime_s`. - **`AnalysisRenderTask(name)`** — **always local** (the only step importing plotstyle/LaTeX). Runs `merge_all` then `render_run`, plus `gallery generate` when `gallery = true`. Target: `<prep_dir>/plots/metadata.yaml`. - **`WorkflowTask`** — `b2luigi.WrapperTask` requiring one `AnalysisRenderTask` per `[[analysis]]`. ### Settings wiring (`run.py`) `result_dir`, `log_dir`, `task_file_dir` from the spec; `batch_system = "htcondor"`; `working_dir = repo_dir`; `env_script`; `executable = [".venv/bin/python"]`. `/ceph` is shared between submit host and workers, so **no `transfer_files`** — result and log dirs must be on `/ceph`. `AnalysisRenderTask` overrides `batch_system = "local"` as a class property, which b2luigi honours per task. GPU requirement strings are ported from `condor-gpu-train-rollout:giant/condor.py::_gpu_requirements` (`TARGET.ProvidesEtpCeph =?= True` ANDed with `GPUs_DeviceName` / `GPUs_GlobalMemoryMb` pins) rather than rewritten. ### Deletions and CLI reduction - `giant/analysis/condor.py`: delete `SubmitConfig`, `_WRAPPER`, `_submit_description`, `_job_walltimes`, `_resolve_giant_executable`, `write_submit`. Keep `prep`, `derive_run_dir`, `RunMeta`, `load_rollout_yaml(s)`, `compute_reduced`/`compute_one`, `merge_one`/`merge_all` — that is the real logic — and rename the module to `giant/analysis/run.py` since nothing in it submits any more. Update the re-exports in `giant/analysis/__init__.py` and its module docstring. - `giant/cli.py`: delete the `analyze submit` command (`cli.py:1680-1756`). `prep`, `compute-one`, `merge-one`, `list`, `render`, `metrics` stay as primitives. - Do **not** port `train-submit` / `rollout-submit` from `condor-gpu-train-rollout` — the workflow supersedes them, and that branch's `giant/condor.py` is reduced to the requirement-string helpers moved into `giant/workflow/htcondor.py`. This is a decision that branch's eventual merge must respect; note it in CLAUDE.md. - `tests/test_condor.py`: drop the `write_submit`/submit-description cases, keep the `prep`/`merge` ones. ### Required change to `giant rollout` `_write_prediction_ref` (`giant/cli.py:188`) writes the sidecar to `<checkpoint.parent>/<random-uuid>.yaml`, which is not a deterministic target. Change it so that **when `--out` is passed explicitly**, the sidecar goes to `out.with_suffix(".yaml")`; the existing uuid-under-the-checkpoint behaviour is kept for the no-`--out` case so ad-hoc runs and the `/ceph` predictions convention are unaffected. Apply the same rule to `giant predict` for consistency. ### Dependency Add `b2luigi>=1.0,<2` under a new `workflow` optional-dependency extra in `pyproject.toml` (it pulls `luigi` + `tenacity`), and include `giant[workflow]` in the `dev` extra. Document `uv sync --extra cpu --extra workflow` in CLAUDE.md. ## Files - **new**: `giant/workflow/{__init__,spec,htcondor,tasks,run}.py`, `tests/test_workflow_spec.py`, `tests/test_workflow_tasks.py`, an example `configs/workflow_example.toml` - **modified**: `giant/cli.py` (delete `analyze submit`, add `workflow` sub-app, sidecar path rule), `giant/analysis/condor.py` → `giant/analysis/run.py`, `giant/analysis/__init__.py`, `tests/test_condor.py`, `pyproject.toml`, `CLAUDE.md`, `README.md` ## Verification 1. `uv sync --extra cpu --extra workflow --extra dev`, then `uv run pytest`, `uv run ruff check .`, `uv run ty check .`. 2. `giant workflow run configs/workflow_example.toml --mode dry-run` on a local machine — asserts the whole DAG resolves and prints the tasks that would run; exit code 1 means work pending, 0 means everything already done. 3. `--mode show-output` on the same spec — eyeball that every target path is where the plan says it is (green = exists, red = missing). 4. Local end-to-end smoke on a tiny parquet (a few hundred events, 3 epochs, 1 rollout, `chunks = 1`, `batch_system = "local"`): confirm three epoch dirs each with a `last.pt`, the published `best.pt`/concatenated `metrics.csv`, `rollout.parquet`/`rollout.yaml`, `reduced_partial/*.json`, and `plots/metadata.yaml`, and that a second invocation is a no-op. 5. Per-epoch chaining is equivalent to a single run: train the same tiny config for 3 epochs in one `giant train` invocation with a fixed seed and diff its `metrics.csv` against the workflow's concatenated one — the per-epoch losses should match (modulo the shuffle-seed question above). 6. Idempotency/restart: delete the epoch-3 output dir and re-run — only epoch 3 onwards should re-execute; separately delete one `reduced_partial/<id>__<chunk>.json` and confirm exactly that one compute task re-runs, then render. 7. On a portal machine, `giant workflow run <real spec> --batch --workers 20` against the real dataset, optionally with `luigid` running for the progress UI; verify with `condor_q -batch <job_name>` that the (plot, chunk) job count matches `len(catalog_ids()) × chunks` (minus the chunk collapse for the five non-chunkable specs), and that render only fires after the last compute job succeeds.
lars added the architecture label 2026-08-26 10:22:10 +02:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
architecture
bug
Silent/wrong behavior, no crash
 experiment
feature
New feature matching the v0.3 redesign
 investigation
modularity
Making the network genuinely pluggable
 performance
phase2
validation
wontfix
Not being fixed right now
No Label architecture
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
lars (Lars Bogner)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lars/giant#83
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?