Add b2luigi dependency and workflow prerequisites (gitea #83)
Groundwork for the b2luigi pipeline orchestration in gitea #83, split out so the workflow package itself lands as a self-contained change: - new `workflow` optional-dependency extra (b2luigi, which pulls luigi + tenacity), included in `dev`. - deterministic rollout/predict sidecar path: with an explicit `--out`, the YAML goes to `out.with_suffix(".yaml")` instead of a uuid-named file under the checkpoint directory, so a workflow task can declare it as a target. The uuid behaviour is kept for the no-`--out` case, leaving ad-hoc runs and the /ceph predictions convention untouched. - epoch-aware shuffle seeding in StreamingStepsDataset (`seed` + `set_epoch`, the DistributedSampler convention). Shuffling previously drew from the global numpy state, which `run_train_job` reseeds from `train.seed` at process start — so a one-epoch-per-job chain would have replayed the same batch order every epoch. Seeding from `(seed, epoch, worker_id)` makes epoch k's order identical whether it runs inside one long `giant train` or as its own resumed job. The val-split seed is untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
+6
-1
@@ -28,7 +28,7 @@ dev = [
|
||||
"ty>=0.0.50,<0.1",
|
||||
"bump-my-version>=1.2,<2",
|
||||
"git-cliff>=2,<3",
|
||||
"giant[convert,analysis,geometry,wandb]",
|
||||
"giant[convert,analysis,geometry,wandb,workflow]",
|
||||
]
|
||||
geometry = [
|
||||
"scikit-learn>=1.4,<2",
|
||||
@@ -49,6 +49,11 @@ analysis = [
|
||||
# `giant analyze render` step imports it; compute workers never do.
|
||||
"plotstyle>=1.0.0",
|
||||
]
|
||||
# b2luigi pulls luigi + tenacity; the only sanctioned way to chain a
|
||||
# multi-step pipeline (see giant/workflow/).
|
||||
workflow = [
|
||||
"b2luigi>=1.0,<2",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
giant = "giant.cli:app"
|
||||
|
||||
Reference in New Issue
Block a user