One workflow TOML now parameterises a whole experiment and `giant workflow
run <spec.toml>` turns it into a b2luigi DAG whose targets are files on
/ceph: nothing already produced is recomputed, every step waits for its
inputs, and HTCondor submission/polling is b2luigi's job.
- spec.py: workflow TOML -> frozen dataclasses with name-uniqueness and
cross-reference validation, unknown keys rejected the way giant.config
rejects them, and a short spec_hash per task that folds in its transitive
parents — so an edited spec re-runs exactly the affected subtree.
- htcondor.py: the CPU/GPU submit settings. The GPU requirement strings
(ProvidesEtpCeph + optional device/memory pins) are ported from the
condor-gpu-train-rollout branch rather than rewritten.
- tasks.py: DatasetTask, WarmCacheTask, GeometryOracleTask, TrainEpochTask
(one short GPU job per epoch, chained via --resume, which the training
loop already supports unchanged), TrainTask (publishes best.pt/last.pt and
a concatenated metrics.csv so downstream never sees the epoch fan-out),
RolloutTask, AnalysisPrepTask, AnalysisComputeTask (one job per plot x
chunk, walltime sized from run_meta.json at submit time), AnalysisRenderTask
(always local — the only step importing plotstyle/LaTeX), WorkflowTask.
Task bodies call the existing entry points; none of them reimplement
anything.
- run.py + `giant workflow run`: settings wiring and the script b2luigi
re-executes on workers. add_filename_to_cmd is off because b2luigi passes
only the script's basename, and --spec is forwarded via
task_cmd_additional_args so a worker resolves the identical task graph.
configs/workflow_example.toml is the documented starting point.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A fixed comparison point for future architecture variants, so each
experimental axis (routed trunk, WGAN generators, attention history,
shared conditioning) is a single edit away from one known config.
flow/flow autoregressive, hidden_dim 512 / 6 blocks per stage, physical
conditioning, no router, 7.70M params. Chosen by ranking the five runs in
analysis_runs/ by mean Jensen-Shannon divergence against the Geant4
reference: unrouted flow wins (0.172) over routed flow (0.197/0.200) and
both WGAN runs (0.218/0.234), with the lead concentrated in per-event
total deposited energy and the per-PDG marginals.
batch_size 36864 is sized for one L40S on deepthought2 from a measured
linear fit of this config's training step (reserved MiB = 0.9736 * bs +
115), giving ~36 GiB, 78% of the card.
The comments record two measured facts that are easy to get wrong:
WGAN is slower to *train* than flow (n_critic plus the gradient-penalty
double-backward), its advantage being inference-only; and
sample_secondaries_ar loops over all k_max slots unconditionally rather
than short-circuiting on n_sec, which is what makes the autoregressive
decoder the dominant cost on both axes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scaled-up variant of the already-trained wgan_h128_b4_physical.toml
benchmark config, same training recipe (mode/epochs/lr/warmup),
increased model capacity.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the 10-expert EnergyRouter + physical-conditioning benchmark
config with the new opt-in gumbel combine weights, isolating the
learn_centers/learn_temperature axis: none, centers only, and
centers+temperature.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lars