lars
a482b04761
One workflow TOML now parameterises a whole experiment and `giant workflow run <spec.toml>` turns it into a b2luigi DAG whose targets are files on /ceph: nothing already produced is recomputed, every step waits for its inputs, and HTCondor submission/polling is b2luigi's job. - spec.py: workflow TOML -> frozen dataclasses with name-uniqueness and cross-reference validation, unknown keys rejected the way giant.config rejects them, and a short spec_hash per task that folds in its transitive parents — so an edited spec re-runs exactly the affected subtree. - htcondor.py: the CPU/GPU submit settings. The GPU requirement strings (ProvidesEtpCeph + optional device/memory pins) are ported from the condor-gpu-train-rollout branch rather than rewritten. - tasks.py: DatasetTask, WarmCacheTask, GeometryOracleTask, TrainEpochTask (one short GPU job per epoch, chained via --resume, which the training loop already supports unchanged), TrainTask (publishes best.pt/last.pt and a concatenated metrics.csv so downstream never sees the epoch fan-out), RolloutTask, AnalysisPrepTask, AnalysisComputeTask (one job per plot x chunk, walltime sized from run_meta.json at submit time), AnalysisRenderTask (always local — the only step importing plotstyle/LaTeX), WorkflowTask. Task bodies call the existing entry points; none of them reimplement anything. - run.py + `giant workflow run`: settings wiring and the script b2luigi re-executes on workers. add_filename_to_cmd is off because b2luigi passes only the script's basename, and --spec is forwarded via task_cmd_additional_args so a worker resolves the identical task graph. configs/workflow_example.toml is the documented starting point. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
69 lines
2.0 KiB
TOML
69 lines
2.0 KiB
TOML
# Example GIANT workflow spec — `giant workflow run configs/workflow_example.toml`.
|
|
#
|
|
# One file parameterises a whole experiment: the datasets, the geometry oracle,
|
|
# N trainings, N rollouts, and the analyses comparing them. Every task's output
|
|
# directory carries a hash of its resolved sub-spec (plus its parents), so
|
|
# editing anything here re-runs exactly the affected subtree and nothing else.
|
|
#
|
|
# result_dir/log_dir must be visible from both the submit host and the workers
|
|
# (i.e. on /ceph) — there is deliberately no HTCondor file transfer.
|
|
|
|
[workflow]
|
|
name = "baseline-vs-router"
|
|
result_dir = "/ceph/lbogner/workflows/baseline-vs-router"
|
|
log_dir = "/ceph/lbogner/workflows/baseline-vs-router/logs"
|
|
|
|
[condor]
|
|
accounting_group = "cms"
|
|
repo_dir = "/work/lbogner/giant" # also b2luigi's working_dir
|
|
env_script = "/work/lbogner/giant/condor_env.sh"
|
|
docker_image_cpu = "cverstege/alma9-gridjob"
|
|
docker_image_gpu = "mschnepf/slc7-condocker"
|
|
remote = true
|
|
|
|
[dataset]
|
|
steps = "/ceph/lbogner/geant_steps/train/" # training data
|
|
reference = "/ceph/lbogner/geant_steps/holdout/" # rollout seeds + analysis truth
|
|
|
|
[geometry]
|
|
method = "slab"
|
|
subsample = 500_000
|
|
|
|
[[train]]
|
|
name = "baseline"
|
|
config = "configs/baseline.toml"
|
|
epochs = 200
|
|
epochs_per_job = 1 # one short GPU job per epoch, chained
|
|
request_gpus = 1
|
|
gpu_memory_mb = 20000
|
|
overrides = { lr = 3e-4 } # `giant train` flag names
|
|
|
|
[[train]]
|
|
name = "router-balanced"
|
|
config = "configs/router.toml"
|
|
epochs = 200
|
|
epochs_per_job = 1
|
|
request_gpus = 1
|
|
gpu_memory_mb = 20000
|
|
|
|
[[rollout]]
|
|
name = "baseline"
|
|
train = "baseline" # -> [[train]].name
|
|
n_events = 2000
|
|
energy_cutoff = 0.1
|
|
|
|
[[rollout]]
|
|
name = "router-balanced"
|
|
train = "router-balanced"
|
|
n_events = 2000
|
|
energy_cutoff = 0.1
|
|
|
|
[[analysis]]
|
|
name = "baseline-vs-router"
|
|
rollouts = ["baseline", "router-balanced"]
|
|
chunks = 32
|
|
energy_bins = 4
|
|
bins = 50
|
|
top_pdg = 6
|
|
gallery = true
|