Add b2luigi dependency and workflow prerequisites (gitea #83)

Groundwork for the b2luigi pipeline orchestration in gitea #83, split out
so the workflow package itself lands as a self-contained change:

- new `workflow` optional-dependency extra (b2luigi, which pulls luigi +
  tenacity), included in `dev`.
- deterministic rollout/predict sidecar path: with an explicit `--out`, the
  YAML goes to `out.with_suffix(".yaml")` instead of a uuid-named file under
  the checkpoint directory, so a workflow task can declare it as a target.
  The uuid behaviour is kept for the no-`--out` case, leaving ad-hoc runs and
  the /ceph predictions convention untouched.
- epoch-aware shuffle seeding in StreamingStepsDataset (`seed` +
  `set_epoch`, the DistributedSampler convention). Shuffling previously drew
  from the global numpy state, which `run_train_job` reseeds from
  `train.seed` at process start — so a one-epoch-per-job chain would have
  replayed the same batch order every epoch. Seeding from
  `(seed, epoch, worker_id)` makes epoch k's order identical whether it runs
  inside one long `giant train` or as its own resumed job. The val-split
  seed is untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
2026-08-26 11:43:36 +02:00
parent b66574877b
commit cd73aa2966
6 changed files with 318 additions and 9 deletions
+6 -1
View File
@@ -28,7 +28,7 @@ dev = [
"ty>=0.0.50,<0.1",
"bump-my-version>=1.2,<2",
"git-cliff>=2,<3",
"giant[convert,analysis,geometry,wandb]",
"giant[convert,analysis,geometry,wandb,workflow]",
]
geometry = [
"scikit-learn>=1.4,<2",
@@ -49,6 +49,11 @@ analysis = [
# `giant analyze render` step imports it; compute workers never do.
"plotstyle>=1.0.0",
]
# b2luigi pulls luigi + tenacity; the only sanctioned way to chain a
# multi-step pipeline (see giant/workflow/).
workflow = [
"b2luigi>=1.0,<2",
]
[project.scripts]
giant = "giant.cli:app"