perf: replace pandas with polars in the setup-stage scan

giant.pipeline.run_setup_stage (used by both giant train and dwarf
warm-cache) previously opened and fully read each parquet file 4-6
separate times via pandas, with per-row Python loops padding the
secondary list columns on every chunk of the normalizer-fitting pass.

- giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list
  padding is now a single vectorized polars expression instead of a
  per-row Python loop (including a .iloc[i] loop for directions).
- giant/data/scan.py (new): a fused metadata scan answering the event
  index, pdg/material vocab, process counts, and pooled-pdg counts in
  one pass per file instead of one pass per section. Frequency-ranking
  ties are now an explicit (-count, first_seen) contract instead of an
  accident of pandas' value_counts iteration order.
- giant/pipeline.py: run_setup_stage restructured to consult the cache
  for every section first, then issue one combined scan request for
  whatever's missing.
- giant/geometry.py: ported the one remaining pandas groupby to polars.
- pyproject.toml: polars promoted to a core dependency, pandas moved
  to dev (only test fixtures still use it).
- giant/tools/profile_setup_scan.py (new): synthetic-data benchmark
  for this scan, mirroring profile_analysis_costs.py's pattern.

Also fixes a real deadlock this surfaced: DataLoader worker
subprocesses fork() on Linux, and polars' native thread pool doesn't
survive a fork — a worker touching polars after the parent already had
hangs instantly. giant/pipeline.py's train/val DataLoaders now use
multiprocessing_context="spawn" whenever num_workers>0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC

.bumpversion.toml

@@ -1,5 +1,5 @@
 [tool.bumpversion]
-current_version = "0.4.0"
+current_version = "0.3.16"
 parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
 serialize = ["{major}.{minor}.{patch}"]
 search = "{current_version}"

CLAUDE.md

@@ -9,7 +9,6 @@ uv sync --extra cpu                # install dependencies with CPU-only torch (s
 uv sync --extra cuda               # install dependencies with CUDA 11.8 torch
 uv sync --extra cpu --extra dev    # add dev extras (pytest, ruff, ty, bump-my-version, git-cliff, + all runtime extras)
 uv sync --extra cpu --extra geometry  # add scikit-learn for the geometry oracle (giant rollout)
-uv sync --extra cpu --extra workflow  # add b2luigi for `giant workflow` pipeline orchestration
 pytest               # run tests
 giant new-run --hidden-dim 512 --lr 3e-4        # scaffold a config.toml + run dir ahead of training
 giant train path/to/steps.parquet               # train (defaults: stage 1 flow, stage 2 wgan + autoregressive)
@@ -19,9 +18,8 @@ giant train path/to/steps.parquet --router --router-type energy  # MoE routing t
 giant model summary --config config.toml         # build-only: parameter counts + which config keys actually bite
 giant predict path/to/steps.parquet --checkpoint ckpt/best.pt   # per-step predictions
 giant rollout path/to/steps.parquet --checkpoint ckpt/best.pt --geometry oracle.pkl  # full showers
-giant workflow run spec.toml --batch --workers 20          # whole pipeline (cache-warm -> train -> rollout -> analysis)
-giant analyze prep rollout.yaml --chunks 32                # lay out an analysis run dir (compute jobs come from the workflow)
-giant analyze render <run_dir> --gallery                   # render PDFs + HTML gallery (run_dir from prep)
+giant analyze submit rollout.yaml --accounting-group cms   # parallel rollout-vs-reference analysis on HTCondor
+giant analyze render <run_dir> --gallery                   # render PDFs + HTML gallery (run_dir from prep/submit)
 giant analyze metrics <train_run_dir>                      # training-progress plots from metrics.csv
 dwarf --help                                     # dataset/tooling CLI: convert, migrate, bump-gen,
                                                   # bump-schema, status, update-manifest, create-manifest,
@@ -53,7 +51,7 @@ Work on this repo happens across three kinds of machine:
 
 - **Local dev machines** (laptop + desktop, identical): repo at `~/Programming/giant`, no access to `/ceph` — datasets, training results, and models aren't reachable here.
 - **Portal machines** (`portal1`, `deepthought`, `deepthought2`, `bms1`, `bms2`, `bms3`): repo lives under `/work`, and `/ceph` holds ROOT/parquet files and trained models. **These are shared with other users** — stay strictly within `/work/lbogner` and `/ceph/lbogner`, and keep resource usage to roughly a quarter of CPU/RAM and a single GPU so as not to disturb other users' jobs.
-- **HTCondor worker nodes**: never run or SSH onto these directly — the only sanctioned path is `giant workflow run <spec.toml> --batch` (b2luigi, see the Workflow section), which submits and polls every job. `/ceph` is available there; `/work` is only sometimes mounted, depending on the node.
+- **HTCondor worker nodes**: never run or SSH onto these directly — the only sanctioned path is submitting jobs through condor (`giant analyze submit`, and the in-progress remote-GPU train/rollout submission on `condor-gpu-train-rollout`). `/ceph` is available there; `/work` is only sometimes mounted, depending on the node.
 
 ## Architecture
 
@@ -101,9 +99,7 @@ Secondary energies are a **stick-breaking partition of the `e_sec` budget** from
 
 **Analysis** (`giant/analysis/`, `giant analyze` CLI): a lean, streaming rollout-vs-reference plotting pipeline that compares one or more autoregressive `giant rollout` runs against a single held-out miniCaloSim reference steps file shared by all of them, and produces publication-styled PDFs assembled into an HTML gallery — one distinctly colored series per rollout, one reference line/panel. It exploits the fact that rollout output and a raw reference file share a world-frame physical column subset under identical names (`pre_*`/`post_*`/`edep`/`step_length`/`pdg`/`material`/`event_id`), so no ALR/local-frame decode is needed — everything is world-frame mm/MeV. Structure: `sources.py` (canonical LazyFrames + `RolloutSpec`/`Side` — a rollout's opened frames + per-checkpoint diagnostic inputs — + synthetic-termination-row filtering + the secondary view, which is `generation>0 & step_no==0` rollout tracks vs exploded `sec_*_list` reference columns), `variables.py` (the per-step value expressions shared by range sizing and the plot registry), `reduce.py` (the streaming primitives — a single `hist1d` `group_by([group,bin]).len()` pass, per-event scalars, edep-weighted depth/transverse profiles, species share, leakage), `grouping.py`/`context.py` (fixed bin edges + energy-quantile/pdg/material group sets resolved once by `prep` into `shared.json` over the union of the reference and every rollout, so every compute job is one pass with no range scan), `reduced.py` (`Partial`/`Reduced` — the compact self-describing JSON a compute job emits), `catalog.py` (the declarative `PlotSpec` registry — marginals × {overall,energy,pdg,material}, per-event totals, shower profiles/containment, species/leakage, secondaries, distance summaries, router and type-embedding diagnostics; `giant analyze list` prints every id), `runtime_estimate.py` (per-(plot, chunk) walltime estimates for the submit description), and `render.py` (the only module importing ETPlot's `plotstyle`/LaTeX; dispatches on `Reduced.kind`, writes PDFs + `metadata.yaml`; each rollout gets a stable `ps.get_color(i)` slot by its position in `series`, the reference always draws in one fixed dashed-ink style). `Bundle.rollouts` is a name-keyed dict of `Side`, and every `compute_partial`/`finalize` builds a `Reduced.payload["series"]` dict keyed the same way, with `payload["reference"]` as the one distinguished non-rollout entry. The heatmap-shaped specs (`marginal_distance_summary`, `sec_count_per_step_by_species` — the latter also drawing the reference as its own panel) and the checkpoint-bound diagnostics (`router_gating.py`, `type_embedding_distance.py`) are inherently one-matrix/one-checkpoint per rollout, so they render as one panel per rollout instead of one line/bar per rollout.
 
-**Input is one or more `giant rollout` YAML sidecars** (`run.py:load_rollout_yamls`): each YAML's `output`/`dataset` keys name its rollout parquet and seed file (= the reference truth); every supplied YAML must resolve to the same `dataset`, checked up front with a clear error otherwise (the premise is "N candidates vs one ground truth"). Each rollout's series name comes from a repeated `--label` CLI flag, else the YAML stem (N>1), else `"rollout"` (a single YAML). `prep` creates a **run directory** (`<cwd>/analysis_runs/analysis_<id>/` by default, `--run-dir` to override) holding `shared.json`, `run_meta.json` (`RunMeta.rollouts: list[{name,path,plot_meta}]`, insertion order = CLI order = every plot's series order), `reduced_partial/`, `reduced/`, `plots/`. **Compute/merge/render split:** `giant analyze prep a.yaml [b.yaml ...] --chunks N` records `N` in `run_meta.json`, and the workflow's `AnalysisComputeTask` runs one HTCondor job per (plot, chunk) pair (`compute-one --id --chunk --run-dir`, polars/numpy only — no LaTeX on workers), each streaming over an `event_id`-disjoint slice (`event_id % N == chunk`) of the reference **and every rollout** and writing a small `reduced_partial/<id>__<chunk>.json`; every `PlotSpec` splits into a `compute_partial`/`finalize` pair so chunks can be summed/concatenated back per rollout (`chunkable=False` specs — the checkpoint-bound diagnostics, already bounded/subsampled — always run as a single chunk). The local `giant analyze render <run_dir>` first joins every plot's chunk partials into `reduced/<id>.json` (`merge_all`, a no-op join when `N=1`; `merge-one` does a single plot for debugging), then turns those into the styled PDF/gallery tree. `giant analyze metrics <train_run_dir>` is a separate, unrelated entry point: training-progress plots straight from a run's `metrics.csv`.
-
-**Workflow orchestration** (`giant/workflow/`, `giant workflow run` CLI): b2luigi is the **only sanctioned way to run a multi-step pipeline**; `giant`/`dwarf` are single-step primitives the tasks invoke. One workflow TOML (`configs/workflow_example.toml`) parameterises a whole experiment — `[workflow]`/`[condor]`/`[dataset]`/`[geometry]` plus repeated `[[train]]`/`[[rollout]]`/`[[analysis]]` tables, each cross-referenced by name — and `spec.py` parses it into frozen dataclasses, rejecting unknown keys and dangling references. Every task's output directory is `<result_dir>/<kind>/name=<name>/spec_hash=<hash>/…`, where the 8-hex `spec_hash` covers that task's resolved sub-spec **and its transitive parents**, so an edited spec re-runs exactly the affected subtree instead of silently reusing stale outputs. The DAG (`tasks.py`): `DatasetTask` (external, fails fast if `/ceph` isn't mounted) → `WarmCacheTask` / `GeometryOracleTask` → `TrainEpochTask(name, milestone)` → `TrainTask` → `RolloutTask` → `AnalysisPrepTask` → `AnalysisComputeTask(name, plot_id, chunk)` → `AnalysisRenderTask` → `WorkflowTask`. Training is fanned out into **one short GPU job per epoch** (`epochs_per_job` trades queue waits back), chained by `--resume` on the previous job's `last.pt` — the loop already handles that unchanged — and `TrainTask` republishes `best.pt`/`last.pt`/a concatenated `metrics.csv` so nothing downstream sees the fan-out. `StreamingStepsDataset.set_epoch` and `config.epoch_seed` (both applied per epoch by `training/loop.py`) derive the batch order and the global RNG state from `(seed, epoch)`, so epoch *k* is bit-identical either way — verified by diffing a chained run's `metrics.csv` against a single 3-epoch `giant train`. `AnalysisRenderTask` is always local (the only step importing plotstyle/LaTeX); `htcondor.py` holds the CPU/GPU submit settings, with the GPU requirement strings (`TARGET.ProvidesEtpCeph` + device/memory pins) ported from the `condor-gpu-train-rollout` branch. `run.py` is the script b2luigi re-executes on workers (`--spec` forwarded via `task_cmd_additional_args`, so a worker resolves the identical graph); `giant workflow run` is a thin exec of it. Needs `uv sync --extra cpu --extra workflow`.
+**Input is one or more `giant rollout` YAML sidecars** (`condor.py:load_rollout_yamls`): each YAML's `output`/`dataset` keys name its rollout parquet and seed file (= the reference truth); every supplied YAML must resolve to the same `dataset`, checked up front with a clear error otherwise (the premise is "N candidates vs one ground truth"). Each rollout's series name comes from a repeated `--label` CLI flag, else the YAML stem (N>1), else `"rollout"` (a single YAML). `prep` creates a **run directory** (`<cwd>/analysis_runs/analysis_<id>/` by default, `--run-dir` to override) holding `shared.json`, `run_meta.json` (`RunMeta.rollouts: list[{name,path,plot_meta}]`, insertion order = CLI order = every plot's series order), `reduced_partial/`, `reduced/`, `plots/`. **Compute/merge/render split:** `giant analyze submit a.yaml [b.yaml ...] --chunks N` runs `prep` (recording `N` in `run_meta.json`) then submits one HTCondor job per (plot, chunk) pair (`compute-one --id --chunk --run-dir`, polars/numpy only — no LaTeX on workers), each streaming over an `event_id`-disjoint slice (`event_id % N == chunk`) of the reference **and every rollout** and writing a small `reduced_partial/<id>__<chunk>.json`; every `PlotSpec` splits into a `compute_partial`/`finalize` pair so chunks can be summed/concatenated back per rollout (`chunkable=False` specs — the checkpoint-bound diagnostics, already bounded/subsampled — always run as a single chunk). The local `giant analyze render <run_dir>` first joins every plot's chunk partials into `reduced/<id>.json` (`merge_all`, a no-op join when `N=1`; `merge-one` does a single plot for debugging), then turns those into the styled PDF/gallery tree. `giant analyze metrics <train_run_dir>` is a separate, unrelated entry point: training-progress plots straight from a run's `metrics.csv`.
 
 **Shower rollout** (`giant/rollout.py`, `giant rollout` CLI): autoregressively steps the two-stage model into a full shower, advancing tracks breadth-first (every sweep steps all active tracks once, in `batch_size` chunks, so many tracks share each forward pass). Each primary post-step becomes the next pre-step, secondaries are pushed as new tracks, and per-step `material`/`layer_id` come from a `GeometryOracle` (`giant/geometry.py`, built via `dwarf build-geometry-oracle`) that learns position → (material, layer_id) from data and flags detector escape by nearest-neighbour distance. Tracks terminate on one of the `TERM_*` reasons in `constants.py` (energy cutoff, max steps, escape, natural end, unknown pdg, max tracks); energy is deposited locally on every stop except escape (leakage), so showers conserve energy by construction. `giant/checkpoint_io.py` is the shared checkpoint → ready-to-run-models path used by both `predict` and `rollout`.
 
@@ -127,4 +123,4 @@ v0.2 configs and checkpoints are auto-migrated (`config.migrate_config`, `model.
 
 A sampling-calorimeter (multi-material) dataset track is still open and unblocked, not yet started. See the knowledge base (`/home/lars/knowledge-base/meta/roadmap.md`).
 
-**Condor-submitted GPU training/rollout (`condor-gpu-train-rollout` branch, superseded):** its goal — moving `giant train`/`giant rollout` off the shared portal GPU dev machines onto remote-GPU HTCondor submission — is now met by the b2luigi workflow above. Its `train-submit`/`rollout-submit` commands are deliberately **not** ported and must not be revived when that branch is eventually merged; the only part that survived is `_gpu_requirements`, which moved into `giant/workflow/htcondor.py`.
+**Condor-submitted GPU training/rollout (in progress, `condor-gpu-train-rollout` branch, not yet merged):** moves `giant train`/`giant rollout` off the shared portal GPU dev machines (see Compute environment) onto remote-GPU HTCondor submission on TOpAS/NEMO2. Partway between "needs major features" and feature-complete — not ready to merge yet.

README.md

@@ -110,15 +110,10 @@ giant/
 │   │   ├── reduced.py      #   Partial/Reduced — the compact JSON a compute job emits
 │   │   ├── catalog.py      #   declarative PlotSpec registry (`giant analyze list`)
 │   │   ├── router_gating.py / type_embedding_distance.py   # checkpoint-bound diagnostics
-│   │   ├── runtime_estimate.py  #   per-(plot, chunk) walltime estimates for the job requests
-│   │   ├── run.py          #   prep / compute-one / merge plumbing
+│   │   ├── runtime_estimate.py  #   per-(plot, chunk) walltime estimates for submit
+│   │   ├── condor.py       #   prep / compute-one / merge / submit-description plumbing
 │   │   └── render.py       #   PDFs + HTML gallery (only module importing plotstyle/LaTeX)
-│   ├── workflow/           # b2luigi pipeline orchestration (`giant workflow run spec.toml`)
-│   │   ├── spec.py         #   workflow TOML -> frozen dataclasses, validation, per-task spec hashes
-│   │   ├── htcondor.py     #   CPU/GPU submit settings (docker image, +RemoteJob, GPU requirements)
-│   │   ├── tasks.py        #   the task graph: cache-warm -> train (one job/epoch) -> rollout -> analysis
-│   │   └── run.py          #   the script b2luigi re-executes on every worker
-│   └── cli.py              # `giant train` / `new-run` / `model summary` / `predict` / `rollout` / `analyze` / `workflow`
+│   └── cli.py              # `giant train` / `new-run` / `model summary` / `predict` / `rollout` / `analyze`
 ├── giant/tools/                        # dataset/tooling logic, unified under the `dwarf` CLI (`dwarf --help`)
 │   ├── dwarf.py                        # Typer app: convert, migrate, bump-gen, bump-schema, status,
 │   │                                    #   update-manifest, create-manifest, make-root,
@@ -145,10 +140,9 @@ uv sync --extra cpu --extra geometry  # add scikit-learn, for `dwarf build-geome
 uv sync --extra cpu --extra analysis  # matplotlib/polars/plotstyle, for `giant analyze render`
 uv sync --extra cpu --extra convert   # uproot/awkward/polars, for `dwarf convert`
 uv sync --extra cpu --extra wandb     # W&B logging (`giant train --wandb`)
-uv sync --extra cpu --extra workflow  # b2luigi, for `giant workflow run`
 ```
 
-The `dev` extra pulls in `convert`, `analysis`, `geometry`, `wandb` and `workflow` as well.
+The `dev` extra pulls in `convert`, `analysis`, `geometry` and `wandb` as well.
 
 `cpu` and `cuda` are mutually exclusive — pick one to select the torch build (pinned to 2.3.x). Plain `uv sync` installs no torch at all. See `CLAUDE.md` for details.
 
@@ -185,29 +179,17 @@ Config-file-only knobs (no CLI flag — use `--config config.toml`): `stage2_mod
 - `giant analyze` — deeper rollout-vs-reference diagnostics (marginals by energy/pdg/material, per-event totals, shower profiles, species share, leakage, secondaries):
 
 ```bash
-giant analyze prep rollout.yaml --chunks 8                 # lay out the run directory
-giant analyze prep a.yaml b.yaml --label flow --label wgan  # N rollouts vs one shared reference
+giant analyze submit rollout.yaml --accounting-group cms   # prep + one HTCondor job per plot × chunk (compute only)
+giant analyze submit a.yaml b.yaml --accounting-group cms --label flow --label wgan  # N rollouts vs one shared reference
 giant analyze render <run_dir> --gallery                   # local: merge chunks, then styled PDFs + HTML gallery (needs LaTeX)
 
 giant analyze list                                         # every catalog plot id
+giant analyze prep rollout.yaml --chunks 8                 # just the run directory, no submission
 giant analyze compute-one --id marginal_edep --run-dir <run_dir> --chunk 0   # what a condor job runs
 giant analyze merge-one --id marginal_edep --run-dir <run_dir>               # merge one plot's chunks (debugging)
 ```
 
-The per-(plot, chunk) compute jobs themselves are submitted by the workflow (below), not by `giant analyze` — these commands are the single-step primitives it calls. `<run_dir>` defaults to `<cwd>/analysis_runs/analysis_<id>` (`--run-dir` overrides it; `prep` prints it). Multiple rollout YAMLs must all name the same reference (`dataset`) file; each renders as its own colored series against one reference line/panel. Compute jobs are polars/numpy only; only `render` needs LaTeX, so it always runs locally.
-
-## Workflow orchestration
-
-Multi-step pipelines run through [b2luigi](https://github.com/belle2/b2luigi) — one spec file describes a whole experiment, and every step's outputs are files on `/ceph` that are only recomputed when their spec (or an upstream one) changes:
-
-```bash
-uv sync --extra cpu --extra workflow
-giant workflow run configs/workflow_example.toml --mode dry-run   # what would run
-giant workflow run configs/workflow_example.toml --mode show-output  # where every output goes
-giant workflow run configs/workflow_example.toml --batch --workers 20  # submit to HTCondor and wait
-```
-
-The spec holds `[workflow]`/`[condor]`/`[dataset]`/`[geometry]` plus repeated `[[train]]`, `[[rollout]]` and `[[analysis]]` tables cross-referenced by name (see `configs/workflow_example.toml`). The task graph is `DatasetTask → WarmCacheTask/GeometryOracleTask → TrainEpochTask… → TrainTask → RolloutTask → AnalysisPrepTask → AnalysisComputeTask(plot, chunk) → AnalysisRenderTask`. Training is split into one short GPU job per epoch (chained by `--resume`), which schedules better on a busy farm and survives preemption; `TrainTask` then publishes one `best.pt`/`last.pt`/`metrics.csv` for everything downstream. Rendering always runs locally, since it is the only step that needs LaTeX.
+`<run_dir>` defaults to `<cwd>/analysis_runs/analysis_<id>` (`--run-dir` overrides it; `prep`/`submit` print it). Multiple rollout YAMLs must all name the same reference (`dataset`) file; each renders as its own colored series against one reference line/panel. Compute jobs are polars/numpy only; only `render` needs LaTeX, so it always runs locally.
 
 Separately, `giant analyze metrics <train_run_dir>` renders training-progress plots (loss/lr/accuracy/grad-norm/router/wgan/throughput) straight from a training run's `metrics.csv`.
 

configs/wgan_ar_stoptoken.toml

@@ -0,0 +1,122 @@
+# GIANT WGAN-GP + AR stop-token variant of configs/baseline.toml.
+#
+# Two roadmap axes, combined into one run: WGAN-GP generators for both
+# stages (unbenchmarked since the 2026-08-03 pre-v0.3.0 failure, which was
+# secondary-species mode collapse — the failure v0.3.0's AR/categorical
+# pivot exists to fix) and the AR stop-token multiplicity mode
+# (stage2_model.n_sec.mode = "stop_token", never benchmarked at all).
+# Everything else is byte-identical to baseline.toml so a rollout compared
+# against baseline's analysis_341dfb14 is attributable to these two axes
+# alone: conditioning (physical/physical), hidden_dim 512 / n_res_blocks 6 /
+# dropout 0.0 per stage, k_max 15, history "markov", teacher_forcing
+# "always", particle_type.target "onehot" (n_classes 32, other_policy
+# "sample"), lr 3e-4, warmup_epochs 3, weight_decay 0.01, ema_decay 0.9999,
+# val_fraction 0.1, num_workers 4, seed 0, validate_steps 10, W&B on.
+#
+# No [stage1_model.wgan] / [stage2_model.wgan] block: the dataclass defaults
+# (noise_dim 64, n_critic 5, gp_weight 10.0, critic_lr 0.0 = inherit
+# train.lr, critic_hidden_dim/critic_n_res_blocks 0 = inherit the stage's
+# 512/6, stage 2's gumbel_tau_start/_end 1.0/0.1) are what the earlier WGAN
+# runs used — writing them out would add keys that don't vary.
+#
+# particle_type.class_weighting stays "none" (the default): config.py's
+# validate_config rejects any other value under stage2_model.generator =
+# "wgan", since that path feeds the type slice to the critic via a
+# straight-through Gumbel relaxation instead of a weighted cross-entropy.
+#
+# Prior WGAN writeup (pre-v0.3.0, describes the failure this run re-tests):
+# /home/lars/knowledge-base/experiments/giant-wgan-physical-rollout-validation.md
+
+[meta]
+# REQUIRED. Without it config.migrate_config reads this file as v0.2 and
+# rewrites it from V02_FIXED_FACTS — silently forcing decoder = "one_shot",
+# particle_type.target = "physical" and the v0.2 default sizes, while still
+# passing validate_config.
+config_version = 3
+
+[conditioning]
+# Physical-property MLPs rather than learned vocab embeddings: computable for
+# any PDG code / material, which is what the held-out-species and
+# held-out-material generalization comparisons need.
+out_dim      = 128
+share_stages = false
+
+# n_layers = 2 rather than the v0.3 default of 1: v0.2's conditioning MLP was
+# always 2 deep (see _migration.V02_FIXED_FACTS), so this keeps the encoder
+# identical to baseline.toml.
+[conditioning.particle]
+type     = "physical"
+emb_dim  = 16
+n_layers = 2
+
+[conditioning.material]
+type     = "physical"
+emb_dim  = 16
+n_layers = 2
+
+[stage1_model]
+generator    = "wgan"
+hidden_dim   = 512
+n_res_blocks = 6
+dropout      = 0.0
+
+[stage2_model]
+# Autoregressive in descending-energy order, as baseline.toml — this variant
+# only swaps the generator (flow -> wgan) and the multiplicity mode
+# (head -> stop_token), not the decoder shape.
+decoder      = "autoregressive"
+generator    = "wgan"
+hidden_dim   = 512
+n_res_blocks = 6
+dropout      = 0.0
+k_max        = 15
+
+[stage2_model.autoregressive]
+history         = "markov"
+teacher_forcing = "always"
+
+[stage2_model.n_sec]
+# EOS-style per-slot stop head on the AR secondary decoder, replacing the
+# n_sec classifier entirely (mutually exclusive — see NSecConfig's
+# docstring in giant/config.py). Requires decoder = "autoregressive" and
+# owner = "stage2" (both already true above/by default); validate_config
+# enforces this.
+mode = "stop_token"
+
+[stage2_model.particle_type]
+target = "onehot"
+# Decoupled from conditioning.particle.emb_dim (gitea #29). 32 classes + the
+# "other" bucket keeps essentially all real secondary species out of "other"
+# without making the head expensive.
+n_classes    = 32
+other_policy = "sample"
+
+[train]
+epochs = 30
+# Halved from baseline's 36864. That figure came from a measured linear fit
+# of the *flow-AR* training step (peak reserved MiB = 0.9736 * batch_size +
+# 115); WGAN invalidates it twice over — each stage gains a critic that by
+# default inherits the stage's own 512/6 body, and gradient_penalty
+# (giant/model/wgan.py, forced fp32 internally) runs a double-backward every
+# batch. 18432 is a conservative choice pending a real memory measurement on
+# this exact config, not a re-derived fit. Throughput is already flat above
+# bs~4096 on the 4070, so this costs occupancy on the L40S, not step
+# efficiency.
+batch_size    = 18432
+lr            = 3e-4
+warmup_epochs = 3
+weight_decay  = 0.01
+ema_decay     = 0.9999
+val_fraction  = 0.1
+num_workers   = 4
+seed          = 0
+# Tightened from baseline's 10: WGANStageTrainer.supports_val_loss = False,
+# and with both stages adversarial there is no per-epoch val loss at all, so
+# validate_every's marginal-KL pass (giant/training/trainers.py's
+# val_objective) is the only comparable-across-epochs best-checkpoint
+# selection signal available. 5 gives 6 evaluations over 30 epochs instead
+# of baseline's 3, at ~6x5000s of extra walltime.
+validate_every = 5
+validate_steps = 10
+wandb          = true
+wandb_project  = "giant"

configs/workflow_example.toml

@@ -1,68 +0,0 @@
-# Example GIANT workflow spec — `giant workflow run configs/workflow_example.toml`.
-#
-# One file parameterises a whole experiment: the datasets, the geometry oracle,
-# N trainings, N rollouts, and the analyses comparing them. Every task's output
-# directory carries a hash of its resolved sub-spec (plus its parents), so
-# editing anything here re-runs exactly the affected subtree and nothing else.
-#
-# result_dir/log_dir must be visible from both the submit host and the workers
-# (i.e. on /ceph) — there is deliberately no HTCondor file transfer.
-
-[workflow]
-name       = "baseline-vs-router"
-result_dir = "/ceph/lbogner/workflows/baseline-vs-router"
-log_dir    = "/ceph/lbogner/workflows/baseline-vs-router/logs"
-
-[condor]
-accounting_group = "cms"
-repo_dir         = "/work/lbogner/giant"   # also b2luigi's working_dir
-env_script       = "/work/lbogner/giant/condor_env.sh"
-docker_image_cpu = "cverstege/alma9-gridjob"
-docker_image_gpu = "mschnepf/slc7-condocker"
-remote           = true
-
-[dataset]
-steps     = "/ceph/lbogner/geant_steps/train/"     # training data
-reference = "/ceph/lbogner/geant_steps/holdout/"   # rollout seeds + analysis truth
-
-[geometry]
-method    = "slab"
-subsample = 500_000
-
-[[train]]
-name           = "baseline"
-config         = "configs/baseline.toml"
-epochs         = 200
-epochs_per_job = 1            # one short GPU job per epoch, chained
-request_gpus   = 1
-gpu_memory_mb  = 20000
-overrides      = { lr = 3e-4 }   # `giant train` flag names
-
-[[train]]
-name           = "router-balanced"
-config         = "configs/router.toml"
-epochs         = 200
-epochs_per_job = 1
-request_gpus   = 1
-gpu_memory_mb  = 20000
-
-[[rollout]]
-name          = "baseline"
-train         = "baseline"     # -> [[train]].name
-n_events      = 2000
-energy_cutoff = 0.1
-
-[[rollout]]
-name          = "router-balanced"
-train         = "router-balanced"
-n_events      = 2000
-energy_cutoff = 0.1
-
-[[analysis]]
-name        = "baseline-vs-router"
-rollouts    = ["baseline", "router-balanced"]
-chunks      = 32
-energy_bins = 4
-bins        = 50
-top_pdg     = 6
-gallery     = true

giant/analysis/__init__.py

@@ -4,7 +4,7 @@ Compares one or more autoregressive ``giant rollout`` runs against a single
 held-out miniCaloSim reference file shared by all of them, producing
 publication-styled comparison plots (one colored series per rollout, one
 reference line) generated in parallel on HTCondor (one job per plot x data
-chunk, compute/merge/render split) — orchestrated by ``giant/workflow``.
+chunk, compute/merge/render split).
 
 Only ``render`` (and the ``render`` CLI path) imports plotstyle/LaTeX; everything
 re-exported here is plotstyle-free so it runs on a compute worker. Import
@@ -12,9 +12,10 @@ re-exported here is plotstyle-free so it runs on a compute worker. Import
 """
 
 from giant.analysis.catalog import build_catalog, catalog_ids, get_spec
-from giant.analysis.run import (
+from giant.analysis.condor import (
     LoadedRollout,
     RunMeta,
+    SubmitConfig,
     compute_one,
     compute_reduced,
     derive_run_dir,
@@ -23,6 +24,7 @@ from giant.analysis.run import (
     merge_all,
     merge_one,
     prep,
+    write_submit,
 )
 from giant.analysis.context import Context, build_context
 from giant.analysis.reduced import Partial, Reduced
@@ -35,6 +37,7 @@ __all__ = [
     "get_spec",
     "LoadedRollout",
     "RunMeta",
+    "SubmitConfig",
     "compute_one",
     "compute_reduced",
     "derive_run_dir",
@@ -43,6 +46,7 @@ __all__ = [
     "merge_all",
     "merge_one",
     "prep",
+    "write_submit",
     "Context",
     "build_context",
     "Partial",

giant/analysis/condor.py

@@ -1,6 +1,4 @@
-"""Analysis run directories: prep, per-(plot, chunk) compute, and merge.
-
-Driven by one or more ``giant rollout`` YAML sidecars.
+"""HTCondor orchestration driven by one or more ``giant rollout`` YAML sidecars.
 
 A rollout writes a YAML sidecar (``giant/cli.py:_write_prediction_ref`` +
 rollout extras) that already names both files we need and carries the run's
@@ -24,16 +22,13 @@ everything out under it:
     <run_dir>/reduced/<id>.json   merged, per plot
     <run_dir>/plots/<family>/<id>.pdf   rendered locally
 
-Job model (one job per (plot, chunk), compute/merge/render split). Job
-submission itself is b2luigi's (``giant/workflow/tasks.py`` — ``AnalysisPrepTask``
-/ ``AnalysisComputeTask`` / ``AnalysisRenderTask``); this module only provides
-the three steps they call:
+Job model (one condor job per (plot, chunk), compute/merge/render split):
 
-1. ``prep`` runs once locally — reads the YAML, resolves the shared
+1. ``prep`` runs once on the submit node — reads the YAML, resolves the shared
    context from a subsample, writes ``shared.json`` + ``run_meta.json``
    (including the run's configured ``n_chunks``).
 2. one job per catalog id x chunk index runs ``giant analyze compute-one
-   --run-dir`` (or ``compute_one`` in-process) on a worker — a single streaming pass over that
+   --run-dir`` on a worker — a single streaming pass over that
    ``event_id``-disjoint chunk, writing ``reduced_partial/<id>__<chunk>.json``
    (polars/numpy only, no LaTeX). Specs marked ``chunkable=False``
    (``PlotSpec``, ``catalog.py``) always run as a single chunk.
@@ -43,15 +38,15 @@ the three steps they call:
    ``reduced/<id>.json``, then renders those into the styled PDF + gallery tree
    (that step imports plotstyle/LaTeX).
 
-Files on ``/ceph`` or ``/work`` are reached directly (see
-``giant/workflow/htcondor.py``); no HTCondor file transfer of the multi-GB
-inputs.
+Files on ``/ceph`` or ``/work`` are reached via ``ProvidesETPResources``; no
+HTCondor file transfer of the multi-GB inputs.
 """
 
 from __future__ import annotations
 
 import json
 import shutil
+import sys
 from collections.abc import Sequence
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -62,6 +57,7 @@ import yaml
 from giant.analysis.catalog import Bundle, catalog_ids, get_spec
 from giant.analysis.context import Context, build_context
 from giant.analysis.reduced import Partial
+from giant.analysis.runtime_estimate import estimate_runtime_s
 from giant.analysis.sources import RolloutSpec, Side, open_side
 
 # Keys copied verbatim from a rollout YAML into each plot's gallery metadata.
@@ -436,3 +432,136 @@ def merge_one(spec_id: str, run_dir: str | Path) -> Path:
 def merge_all(run_dir: str | Path) -> list[Path]:
     """Merge every catalog plot's chunk partials into ``reduced/<id>.json``."""
     return [merge_one(spec_id, run_dir) for spec_id in catalog_ids()]
+
+
+# ---------------------------------------------------------------------------
+# submit description
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class SubmitConfig:
+    run_dir: Path
+    accounting_group: str
+    repo_dir: Path
+    docker_image: str = "cverstege/alma9-gridjob"
+    request_memory_mb: int = 8192
+    request_cpus: int = 1
+    remote: bool = False  # +RemoteJob (grid I/O) vs ProvidesETPResources (local files)
+    n_chunks: int = 1  # per-plot data chunks; ignored for chunkable=False specs
+
+
+_WRAPPER = """#!/bin/bash
+set -euo pipefail
+cd {repo_dir}
+exec {giant_exe} analyze compute-one --id "$1" --chunk "$2" --run-dir {run_dir}
+"""
+
+
+def _submit_description(cfg: SubmitConfig, wrapper: Path, jobs_file: Path) -> str:
+    reqs_attrs = "+RemoteJob = True\n" if cfg.remote else "requirements = TARGET.ProvidesETPResources\n"
+    return (
+        "universe = docker\n"
+        f"docker_image = {cfg.docker_image}\n"
+        f"executable = {wrapper}\n"
+        "arguments = $(plotid) $(chunk)\n"
+        "should_transfer_files = YES\n"
+        "when_to_transfer_output = ON_EXIT\n"
+        f"request_memory = {cfg.request_memory_mb}\n"
+        f"request_cpus = {cfg.request_cpus}\n"
+        "+RequestWalltime = $(walltime)\n"
+        f"accounting_group = {cfg.accounting_group}\n"
+        f"{reqs_attrs}"
+        f"output = {cfg.run_dir}/logs/$(plotid)__$(chunk).out\n"
+        f"error = {cfg.run_dir}/logs/$(plotid)__$(chunk).err\n"
+        f"log = {cfg.run_dir}/logs/condor.log\n"
+        f"queue plotid,chunk,walltime from {jobs_file}\n"
+    )
+
+
+def _job_walltimes(run_dir: Path, ids: list[str], n_chunks: int) -> list[tuple[str, int, int]]:
+    """``(spec_id, chunk, walltime_s)`` for every job, sized from ``run_meta.json``.
+
+    Row counts come from ``prep``'s ``RunMeta.rows_per_chunk``/``total_rows``;
+    ``chunkable=False`` specs (router diagnostics) always use the dataset
+    total since they run as a single job regardless of ``n_chunks``.
+    """
+    meta = RunMeta.load(run_dir / "run_meta.json")
+    jobs: list[tuple[str, int, int]] = []
+    for spec_id in ids:
+        chunkable = get_spec(spec_id).chunkable
+        chunks = range(n_chunks) if chunkable else [0]
+        for chunk in chunks:
+            n_rows = meta.rows_per_chunk[chunk] if chunkable else meta.total_rows
+            jobs.append((spec_id, chunk, estimate_runtime_s(spec_id, n_rows)))
+    return jobs
+
+
+def _resolve_giant_executable(repo_dir: Path) -> Path:
+    """Path to the ``giant`` entry point to bake into the condor wrapper script.
+
+    Prefers the venv currently running this process (``sys.executable``'s
+    sibling ``giant``) so a submit from a non-default venv (e.g. ``--extra
+    cuda`` on a dev box) doesn't silently pick up a different one; falls back
+    to ``repo_dir/.venv/bin/giant`` for the case this is invoked from outside
+    any venv (e.g. a system Python).
+    """
+    active = Path(sys.executable).parent / "giant"
+    if active.exists():
+        return active
+    venv_giant = repo_dir / ".venv" / "bin" / "giant"
+    if not venv_giant.exists():
+        raise FileNotFoundError(
+            f"no `giant` executable found next to {sys.executable} or at "
+            f"{venv_giant} — condor jobs run it directly (no `uv` on the "
+            f"worker image), so run `uv sync --extra cpu` in {repo_dir} "
+            "before submitting."
+        )
+    return venv_giant
+
+
+def write_submit(cfg: SubmitConfig, ids: list[str] | None = None) -> Path:
+    """Write the wrapper script, (plot, chunk) job list, and HTCondor submit
+    description.
+
+    Each catalog id gets ``cfg.n_chunks`` jobs, except ``chunkable=False``
+    specs (the router diagnostics), which always get exactly one regardless of
+    ``cfg.n_chunks``. Every job's ``+RequestWalltime`` is estimated from its
+    chunk's row count (``runtime_estimate.estimate_runtime_s``, requires
+    ``run_meta.json`` from ``prep`` to already carry ``rows_per_chunk``).
+    Returns the submit description path (``<run_dir>/analyze.sub``). Does not
+    submit — call ``condor_submit`` on the returned file.
+
+    ``cfg.n_chunks`` and the run directory's own ``RunMeta.n_chunks`` (fixed by
+    ``prep``, and what ``RunMeta.rows_per_chunk`` was sized against) are two
+    independent values — checked equal up front so a mismatch is a clear error
+    here rather than an ``IndexError`` out of ``_job_walltimes``.
+    """
+    giant_exe = _resolve_giant_executable(cfg.repo_dir)
+
+    ids = ids or catalog_ids()
+    run_dir = cfg.run_dir
+    meta = RunMeta.load(run_dir / "run_meta.json")
+    if cfg.n_chunks != meta.n_chunks:
+        raise ValueError(
+            f"SubmitConfig.n_chunks={cfg.n_chunks} does not match the "
+            f"n_chunks this run directory was prepped with "
+            f"(RunMeta.n_chunks={meta.n_chunks} in {run_dir}/run_meta.json) — "
+            "re-run `prep` with the desired n_chunks, or fix cfg.n_chunks to "
+            "match it."
+        )
+    (run_dir / "logs").mkdir(parents=True, exist_ok=True)
+    (run_dir / "reduced").mkdir(parents=True, exist_ok=True)
+    (run_dir / "reduced_partial").mkdir(parents=True, exist_ok=True)
+
+    wrapper = run_dir / "run_compute.sh"
+    wrapper.write_text(_WRAPPER.format(repo_dir=cfg.repo_dir, giant_exe=giant_exe, run_dir=run_dir))
+    wrapper.chmod(0o755)
+
+    jobs = _job_walltimes(run_dir, ids, cfg.n_chunks)
+    jobs_file = run_dir / "jobs.txt"
+    jobs_file.write_text("\n".join(f"{i},{k},{w}" for i, k, w in jobs) + "\n")
+
+    sub = run_dir / "analyze.sub"
+    sub.write_text(_submit_description(cfg, wrapper, jobs_file))
+    return sub

giant/analysis/render.py

@@ -545,7 +545,7 @@ def render_run(run_dir: str | Path, *, run_gallery: bool = False) -> list[Path]:
     (checkpoint, paths, cutoffs) from ``run_meta.json`` into every plot's
     gallery metadata and renders.
     """
-    from giant.analysis.run import RunMeta, merge_all
+    from giant.analysis.condor import RunMeta, merge_all
 
     run_dir = Path(run_dir)
     merge_all(run_dir)

giant/analysis/runtime_estimate.py

@@ -1,4 +1,4 @@
-"""Per-(plot, chunk) HTCondor walltime estimates for the analysis compute jobs.
+"""Per-(plot, chunk) HTCondor walltime estimates for `giant analyze submit`.
 
 Each catalog spec's compute cost is close to linear in the number of input
 rows a `compute-one` job streams over — every spec is one (or a couple of)

giant/cli.py

@@ -6,7 +6,6 @@ from enum import Enum
 import math
 from pathlib import Path
 import re
-import sys
 from typing import TYPE_CHECKING, Optional, cast
 import uuid as uuid_mod
 
@@ -188,17 +187,8 @@ def _write_prediction_ref(
     out: Path,
     dataset_path: Path,
     comment: str | None = None,
-    explicit_out: bool = False,
 ) -> Path:
-    """Write the YAML sidecar and return its path.
-
-    With an explicit ``--out`` the sidecar sits next to the output file as
-    ``out.with_suffix(".yaml")`` — a *deterministic* path, which is what lets
-    a workflow task (``giant/workflow/tasks.py``) declare it as a target.
-    Without one, the historic uuid-named file under the checkpoint directory
-    is kept, so ad-hoc runs and the ``/ceph`` predictions convention are
-    unaffected.
-    """
+    """Write a YAML sidecar in the checkpoint directory and return its path."""
     import yaml
 
     ref = {
@@ -210,7 +200,7 @@ def _write_prediction_ref(
     }
     if comment is not None:
         ref["comment"] = comment
-    ref_path = out.with_suffix(".yaml") if explicit_out else checkpoint.parent / f"{pred_uuid}.yaml"
+    ref_path = checkpoint.parent / f"{pred_uuid}.yaml"
     ref_path.write_text(yaml.dump(ref, default_flow_style=False, sort_keys=False))
     return ref_path
 
@@ -1158,7 +1148,6 @@ def predict(
     bs = batch_size_value
 
     # --- Output path ---
-    explicit_out = out is not None
     out, dataset_path, pred_uuid = _resolve_prediction_output(data, out)
     out.parent.mkdir(parents=True, exist_ok=True)
     typer.echo(f"output: {out}")
@@ -1378,7 +1367,7 @@ def predict(
     if writer is not None:
         writer.close()
 
-    ref_path = _write_prediction_ref(checkpoint, pred_uuid, out, dataset_path, comment, explicit_out=explicit_out)
+    ref_path = _write_prediction_ref(checkpoint, pred_uuid, out, dataset_path, comment)
     typer.echo(f"reference: {ref_path}")
 
     if skipped:
@@ -1557,7 +1546,6 @@ def rollout(
     seeds = _seed_from_data(files, n_events)
     typer.echo(f"seeded {len(seeds['event_id']):,} shower(s)")
 
-    explicit_out = out is not None
     out, dataset_path, pred_uuid = _resolve_prediction_output(data, out)
     out.parent.mkdir(parents=True, exist_ok=True)
 
@@ -1640,7 +1628,7 @@ def rollout(
 
     l1_summary = l1_dist_collector.summary()
 
-    ref_path = _write_prediction_ref(checkpoint, pred_uuid, out, dataset_path, explicit_out=explicit_out)
+    ref_path = _write_prediction_ref(checkpoint, pred_uuid, out, dataset_path)
     ref = yaml.safe_load(ref_path.read_text())
     ref.update(
         {
@@ -1694,49 +1682,6 @@ def rollout(
     typer.echo(f"reference: {ref_path}")
 
 
-workflow_app = typer.Typer(
-    no_args_is_help=True,
-    help="b2luigi pipeline orchestration: one spec file -> cache-warm, train, rollout, analysis.",
-)
-app.add_typer(workflow_app, name="workflow")
-
-
-@workflow_app.command("run")
-def workflow_run(
-    spec: Annotated[Path, typer.Argument(help="Workflow TOML (see configs/workflow_example.toml)")],
-    batch: Annotated[
-        bool,
-        typer.Option("--batch/--local", help="Submit batch-system tasks to HTCondor, or run everything locally"),
-    ] = False,
-    workers: Annotated[int, typer.Option("--workers", help="Concurrent luigi workers")] = 1,
-    mode: Annotated[
-        str,
-        typer.Option(
-            "--mode",
-            help="run | dry-run (print pending tasks) | show-output (print every target) | remove (delete outputs)",
-        ),
-    ] = "run",
-    scheduler_host: Annotated[Optional[str], typer.Option("--scheduler-host", help="luigid host")] = None,
-    scheduler_port: Annotated[Optional[int], typer.Option("--scheduler-port", help="luigid port")] = None,
-) -> None:
-    """Run a workflow spec end to end (the only sanctioned multi-step entry point).
-
-    A thin exec of `giant/workflow/run.py`, which b2luigi also re-executes on
-    every worker — so there is one documented entry point and one code path.
-    """
-    import subprocess
-
-    script = Path(__file__).resolve().parent / "workflow" / "run.py"
-    cmd = [sys.executable, str(script), "--spec", str(spec), "--workers", str(workers), "--mode", mode]
-    if batch:
-        cmd.append("--batch")
-    if scheduler_host:
-        cmd += ["--scheduler-host", scheduler_host]
-    if scheduler_port:
-        cmd += ["--scheduler-port", str(scheduler_port)]
-    raise typer.Exit(subprocess.run(cmd).returncode)
-
-
 analyze_app = typer.Typer(
     no_args_is_help=True,
     help="Rollout-vs-reference analysis: parallel compute on HTCondor + local render.",
@@ -1867,5 +1812,83 @@ def analyze_metrics(
     typer.echo(f"rendered {len(paths)} plots -> {paths[0].parent if paths else '(nothing to render)'}")
 
 
+@analyze_app.command("submit")
+def analyze_submit(
+    rollout_yamls: Annotated[
+        list[Path],
+        typer.Argument(
+            help="giant rollout YAML sidecar(s). Multiple compare N rollouts against one "
+            "shared reference — every YAML must name the same `dataset`."
+        ),
+    ],
+    accounting_group: Annotated[str, typer.Option("--accounting-group")],
+    label: Annotated[
+        Optional[list[str]],
+        typer.Option(
+            "--label",
+            help="Series name for a rollout YAML, positionally matched to it — give none, "
+            'or exactly one per YAML. Defaults to the YAML stem (or "rollout" for a '
+            "single YAML).",
+        ),
+    ] = None,
+    run_dir: Annotated[
+        Optional[Path],
+        typer.Option(
+            "--run-dir",
+            "-o",
+            help="Override the run directory (default: <cwd>/analysis_runs/analysis_<id>)",
+        ),
+    ] = None,
+    docker_image: Annotated[str, typer.Option("--docker-image")] = "cverstege/alma9-gridjob",
+    request_memory: Annotated[int, typer.Option("--request-memory", help="MB")] = 8192,
+    remote: Annotated[
+        bool,
+        typer.Option("--remote/--local", help="+RemoteJob vs ProvidesETPResources"),
+    ] = False,
+    chunks: Annotated[
+        int,
+        typer.Option(
+            "--chunks",
+            help="Split each plot's data into this many event_id chunks/jobs",
+        ),
+    ] = 1,
+    n_energy_bins: Annotated[int, typer.Option("--energy-bins")] = 4,
+    n_marginal_bins: Annotated[int, typer.Option("--bins")] = 50,
+    top_k_pdg: Annotated[int, typer.Option("--top-pdg")] = 6,
+    dry_run: Annotated[bool, typer.Option("--dry-run", help="Write files but don't condor_submit")] = False,
+) -> None:
+    """prep + write the HTCondor submit description (one job per plot x chunk), then submit."""
+    import subprocess
+
+    from giant.analysis import SubmitConfig, prep, write_submit
+
+    path = prep(
+        rollout_yamls,
+        run_dir,
+        n_chunks=chunks,
+        default_base=Path.cwd() / "analysis_runs",
+        labels=label,
+        n_energy_bins=n_energy_bins,
+        n_marginal_bins=n_marginal_bins,
+        top_k_pdg=top_k_pdg,
+    )
+    cfg = SubmitConfig(
+        run_dir=path,
+        accounting_group=accounting_group,
+        repo_dir=Path.cwd(),
+        docker_image=docker_image,
+        request_memory_mb=request_memory,
+        remote=remote,
+        n_chunks=chunks,
+    )
+    sub = write_submit(cfg)
+    typer.echo(f"run directory: {path}")
+    typer.echo(f"wrote submit description: {sub}")
+    if dry_run:
+        typer.echo("dry-run: not submitting")
+        return
+    subprocess.run(["condor_submit", str(sub)], check=True)
+
+
 if __name__ == "__main__":
     app()

giant/config.py

@@ -1888,19 +1888,6 @@ def resolve_default_out_dir(cfg: dict, base: Path = Path("checkpoints")) -> Path
     return out_dir
 
 
-def epoch_seed(seed: int, epoch: int) -> int:
-    """Per-epoch derivative of the run seed.
-
-    Reseeding the global RNGs from this at the top of every epoch makes epoch
-    *k* draw the same noise whether it runs inside one long `giant train` or
-    as its own resumed job in a per-epoch workflow chain
-    (`giant/workflow/tasks.py:TrainEpochTask`) — without it, a fresh process
-    would restart the stream at epoch 1's state. Mirrors what
-    `StreamingStepsDataset.set_epoch` does for the batch order.
-    """
-    return (int(seed) * 1_000_003 + int(epoch)) % (2**32)
-
-
 def seed_everything(seed: int) -> None:
     import numpy as np
     import torch

giant/data/dataset.py

@@ -97,7 +97,6 @@ class StreamingStepsDataset(IterableDataset):
         mat_topn_map: dict[str, int] | None = None,
         sec_type_class_map: dict | None = None,
         k_max: int = K_MAX,
-        seed: int = 0,
     ) -> None:
         self.files = list(files)
         self._offsets = {path: event_id_offset(i) for i, path in enumerate(self.files)}
@@ -118,35 +117,16 @@ class StreamingStepsDataset(IterableDataset):
         self.mat_topn_map = mat_topn_map
         self.sec_type_class_map = sec_type_class_map
         self.k_max = k_max
-        self.seed = seed
-        self.epoch = 0
-        self._rng = np.random.default_rng()
-
-    def set_epoch(self, epoch: int) -> None:
-        """Select the shuffle stream for `epoch` (the DistributedSampler convention).
-
-        The training loop calls this at the top of every epoch. Shuffling is
-        seeded from `(seed, epoch, worker_id)` rather than the global numpy
-        state so epoch *k*'s batch order is the same whether it runs as epoch
-        *k* of one long `giant train`, or as its own resumed job in a
-        per-epoch workflow chain (`giant/workflow/tasks.py:TrainEpochTask`).
-        Workers are re-forked from this object each epoch (no
-        `persistent_workers`), so setting it here reaches them.
-        """
-        self.epoch = int(epoch)
 
     def __iter__(self):
         worker_info = torch.utils.data.get_worker_info()
         files = self.files
-        worker_id = worker_info.id if worker_info is not None else 0
         if worker_info is not None:
-            files = files[worker_id :: worker_info.num_workers]
-
-        self._rng = np.random.default_rng([self.seed, self.epoch, worker_id])
+            files = files[worker_info.id :: worker_info.num_workers]
 
         if self.shuffle:
             files = list(files)
-            self._rng.shuffle(files)
+            np.random.default_rng().shuffle(files)
 
         buf_cont: list[np.ndarray] = []
         buf_cat: list[np.ndarray] = []
@@ -242,7 +222,7 @@ class StreamingStepsDataset(IterableDataset):
         styp = np.concatenate(buf_type)
 
         if self.shuffle:
-            idx = self._rng.permutation(len(cont))
+            idx = np.random.permutation(len(cont))
             cont, cat, tgt = cont[idx], cat[idx], tgt[idx]
             nsec, sec, proc, styp = nsec[idx], sec[idx], proc[idx], styp[idx]
 

giant/data/loader.py

@@ -1,13 +1,16 @@
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Iterator
+from typing import TYPE_CHECKING, Any, Iterator, Mapping
 
 import numpy as np
-import pandas as pd
+import polars as pl
 import pyarrow.parquet as pq
 
 from giant.constants import K_MAX
 
+if TYPE_CHECKING:
+    from giant.data.scan import ValueStat
+
 # A manifest is a plain text file listing one parquet path per line, used to
 # name a curated subset of files (e.g. a train/holdout pool) without copying
 # or symlinking the underlying parquet files. Lines are resolved relative to
@@ -77,88 +80,75 @@ def find_parquet_files(path: str | Path) -> list[Path]:
     return [p]
 
 
-def _pad_list_col(series: pd.Series, K: int, fill: float = 0.0) -> np.ndarray:
-    """Pad / truncate a list-valued Series to fixed width K → (N, K) float32."""
-    out = np.full((len(series), K), fill, dtype=np.float32)
-    for i, lst in enumerate(series):
-        if lst is not None and len(lst) > 0:
-            n = min(len(lst), K)
-            out[i, :n] = lst[:n]
-    return out
+def _pad_list_column(df: pl.DataFrame, col: str, k: int, fill, dtype: type[pl.DataType] | pl.DataType) -> np.ndarray:
+    """Pad / truncate a list-valued column to fixed width `k` → (N, k) numpy array.
 
-
-def _pad_list_col_int(series: pd.Series, K: int, fill: int = 0) -> np.ndarray:
-    """Pad / truncate a list-valued integer Series to fixed width K → (N, K) int64."""
-    out = np.full((len(series), K), fill, dtype=np.int64)
-    for i, lst in enumerate(series):
-        if lst is not None and len(lst) > 0:
-            n = min(len(lst), K)
-            out[i, :n] = lst[:n]
-    return out
-
-
-def _pad_dir_col(dx: pd.Series, dy: pd.Series, dz: pd.Series, K: int) -> np.ndarray:
-    """Pad three list-valued direction columns → (N, K, 3) float32.
-
-    Padding direction defaults to (0,0,1) (forward) so it is a valid unit vector.
+    Concatenating `k` fill values before truncating to `k` guarantees every
+    row ends up with exactly `k` non-null elements regardless of how short
+    (including empty) or long the original list was, so `list.to_array(k)`
+    (a fixed-size-array dtype) converts to a plain 2D numpy array with a
+    single vectorized expression — no per-row Python loop.
     """
-    N = len(dx)
-    out = np.zeros((N, K, 3), dtype=np.float32)
-    out[:, :, 2] = 1.0
-    for i in range(N):
-        lx, ly, lz = dx.iloc[i], dy.iloc[i], dz.iloc[i]
-        if lx is not None and len(lx) > 0:
-            n = min(len(lx), K)
-            out[i, :n, 0] = lx[:n]
-            out[i, :n, 1] = ly[:n]
-            out[i, :n, 2] = lz[:n]
-    return out
+    fill_tail = pl.lit([fill] * k, dtype=pl.List(dtype))
+    out = df.select(pl.col(col).cast(pl.List(dtype)).list.concat(fill_tail).list.head(k).list.to_array(k).alias("_p"))
+    return out["_p"].to_numpy()
 
 
-def _df_to_dict(df: pd.DataFrame, offset: int = 0, k_max: int = K_MAX) -> dict[str, np.ndarray]:
+def _pad_dir_col(df: pl.DataFrame, dx: str, dy: str, dz: str, k: int) -> np.ndarray:
+    """Pad three list-valued direction columns → (N, k, 3) float32.
+
+    Padding direction defaults to (0, 0, 1) (forward) so it is a valid unit vector.
+    """
+    px = _pad_list_column(df, dx, k, 0.0, pl.Float64)
+    py = _pad_list_column(df, dy, k, 0.0, pl.Float64)
+    pz = _pad_list_column(df, dz, k, 1.0, pl.Float64)
+    return np.stack([px, py, pz], axis=-1).astype(np.float32)
+
+
+def _df_to_dict(df: pl.DataFrame, offset: int = 0, k_max: int = K_MAX) -> dict[str, np.ndarray]:
     has_sec_lists = "sec_E_list" in df.columns
 
     d: dict[str, np.ndarray] = {
         "event_id": _offset_event_id(df["event_id"].to_numpy(), offset),
-        "pdg": df["pdg"].to_numpy(dtype=np.int32),
-        "pre_pos": df[["pre_x", "pre_y", "pre_z"]].to_numpy(dtype=np.float32),
-        "pre_E": df["pre_E"].to_numpy(dtype=np.float32),
-        "pre_dir": df[["pre_dx", "pre_dy", "pre_dz"]].to_numpy(dtype=np.float32),
-        "material": df["material"].to_numpy(dtype=object),
-        "layer_id": df["layer_id"].to_numpy(dtype=np.int32),
-        "n_sec": df["child_track_ids"].apply(len).to_numpy(dtype=np.int32),
-        "e_sec": df["e_sec"].to_numpy(dtype=np.float32),
+        "pdg": df["pdg"].to_numpy().astype(np.int32),
+        "pre_pos": df.select(["pre_x", "pre_y", "pre_z"]).to_numpy().astype(np.float32),
+        "pre_E": df["pre_E"].to_numpy().astype(np.float32),
+        "pre_dir": df.select(["pre_dx", "pre_dy", "pre_dz"]).to_numpy().astype(np.float32),
+        "material": df["material"].to_numpy().astype(object),
+        "layer_id": df["layer_id"].to_numpy().astype(np.int32),
+        "n_sec": df["child_track_ids"].list.len().to_numpy().astype(np.int32),
+        "e_sec": df["e_sec"].to_numpy().astype(np.float32),
         # The physics process that ended the step (e.g. "compt", "phot",
         # "eBrem") — a post-step outcome, so it's a router/classifier
         # supervision label only, never conditioning (see build_process_map*
         # / ProcessRouter). Guarded like has_sec_lists: older parquet
         # conversions predating this column still load fine.
         "process": (
-            df["process"].to_numpy(dtype=object) if "process" in df.columns else np.full(len(df), "", dtype=object)
+            df["process"].to_numpy().astype(object) if "process" in df.columns else np.full(len(df), "", dtype=object)
         ),
-        "step_length": df["step_length"].to_numpy(dtype=np.float32),
-        "post_E": df["post_E"].to_numpy(dtype=np.float32),
-        "delta_e": (df["pre_E"] - df["post_E"]).to_numpy(dtype=np.float32),
-        "edep": df["edep"].to_numpy(dtype=np.float32),
-        "post_dir": df[["post_dx", "post_dy", "post_dz"]].to_numpy(dtype=np.float32),
-        "post_pos": df[["post_x", "post_y", "post_z"]].to_numpy(dtype=np.float32),
+        "step_length": df["step_length"].to_numpy().astype(np.float32),
+        "post_E": df["post_E"].to_numpy().astype(np.float32),
+        "delta_e": (df["pre_E"] - df["post_E"]).to_numpy().astype(np.float32),
+        "edep": df["edep"].to_numpy().astype(np.float32),
+        "post_dir": df.select(["post_dx", "post_dy", "post_dz"]).to_numpy().astype(np.float32),
+        "post_pos": df.select(["post_x", "post_y", "post_z"]).to_numpy().astype(np.float32),
     }
 
     if has_sec_lists:
-        d["sec_E_list"] = _pad_list_col(df["sec_E_list"], k_max)
-        d["sec_pdg_list"] = _pad_list_col_int(df["sec_pdg_list"], k_max)
-        d["sec_dir_list"] = _pad_dir_col(df["sec_dx_list"], df["sec_dy_list"], df["sec_dz_list"], k_max)
+        d["sec_E_list"] = _pad_list_column(df, "sec_E_list", k_max, 0.0, pl.Float64).astype(np.float32)
+        d["sec_pdg_list"] = _pad_list_column(df, "sec_pdg_list", k_max, 0, pl.Int64).astype(np.int64)
+        d["sec_dir_list"] = _pad_dir_col(df, "sec_dx_list", "sec_dy_list", "sec_dz_list", k_max)
 
     return d
 
 
 def load_steps(path: str | Path, offset: int = 0, k_max: int = K_MAX) -> dict[str, np.ndarray]:
-    return _df_to_dict(pd.read_parquet(path), offset=offset, k_max=k_max)
+    return _df_to_dict(pl.read_parquet(path), offset=offset, k_max=k_max)
 
 
 def load_event_ids(path: str | Path, offset: int = 0) -> np.ndarray:
     """Read only the event_id column — cheap scan for split assignment."""
-    ids = pd.read_parquet(path, columns=["event_id"])["event_id"].to_numpy()
+    ids = pl.read_parquet(path, columns=["event_id"])["event_id"].to_numpy()
     return _offset_event_id(ids, offset)
 
 
@@ -170,7 +160,7 @@ def iter_file_chunks(path: str | Path, offset: int = 0, k_max: int = K_MAX) -> I
     module constant for callers that don't care (e.g. Stage-1-only reads)."""
     pf = pq.ParquetFile(path)
     for i in range(pf.num_row_groups):
-        yield _df_to_dict(pf.read_row_group(i).to_pandas(), offset=offset, k_max=k_max)
+        yield _df_to_dict(pl.DataFrame(pf.read_row_group(i)), offset=offset, k_max=k_max)
 
 
 _COND_COLS = [
@@ -190,17 +180,17 @@ _COND_COLS = [
 ]
 
 
-def _cond_df_to_dict(df: pd.DataFrame, offset: int = 0) -> dict[str, np.ndarray]:
+def _cond_df_to_dict(df: pl.DataFrame, offset: int = 0) -> dict[str, np.ndarray]:
     return {
         "event_id": _offset_event_id(df["event_id"].to_numpy(), offset),
-        "pdg": df["pdg"].to_numpy(dtype=np.int32),
-        "pre_pos": df[["pre_x", "pre_y", "pre_z"]].to_numpy(dtype=np.float32),
-        "pre_E": df["pre_E"].to_numpy(dtype=np.float32),
-        "pre_dir": df[["pre_dx", "pre_dy", "pre_dz"]].to_numpy(dtype=np.float32),
-        "material": df["material"].to_numpy(dtype=object),
-        "layer_id": df["layer_id"].to_numpy(dtype=np.int32),
-        "n_sec": df["child_track_ids"].apply(len).to_numpy(dtype=np.int32),
-        "e_sec": df["e_sec"].to_numpy(dtype=np.float32),
+        "pdg": df["pdg"].to_numpy().astype(np.int32),
+        "pre_pos": df.select(["pre_x", "pre_y", "pre_z"]).to_numpy().astype(np.float32),
+        "pre_E": df["pre_E"].to_numpy().astype(np.float32),
+        "pre_dir": df.select(["pre_dx", "pre_dy", "pre_dz"]).to_numpy().astype(np.float32),
+        "material": df["material"].to_numpy().astype(object),
+        "layer_id": df["layer_id"].to_numpy().astype(np.int32),
+        "n_sec": df["child_track_ids"].list.len().to_numpy().astype(np.int32),
+        "e_sec": df["e_sec"].to_numpy().astype(np.float32),
     }
 
 
@@ -208,7 +198,7 @@ def iter_cond_chunks(path: str | Path, offset: int = 0) -> Iterator[dict[str, np
     """Yield conditioning-only row-groups (no post-step columns read from disk)."""
     pf = pq.ParquetFile(path)
     for i in range(pf.num_row_groups):
-        yield _cond_df_to_dict(pf.read_row_group(i, columns=_COND_COLS).to_pandas(), offset=offset)
+        yield _cond_df_to_dict(pl.DataFrame(pf.read_row_group(i, columns=_COND_COLS)), offset=offset)
 
 
 def build_index_maps(
@@ -225,42 +215,28 @@ def build_index_maps(
 def build_index_maps_from_files(
     files: list[Path],
 ) -> tuple[dict[int, int], dict[str, int]]:
-    """Scan only pdg and material columns across all files (2-column read)."""
-    pdg_vals: set[int] = set()
-    mat_vals: set[str] = set()
-    for path in files:
-        df = pd.read_parquet(path, columns=["pdg", "material"])
-        pdg_vals.update(int(v) for v in df["pdg"].unique())
-        mat_vals.update(str(v) for v in df["material"].unique())
+    """Scan only pdg and material columns across all files (fused single-pass scan)."""
+    from giant.data.scan import ScanRequest, scan_metadata
+
+    result = scan_metadata(files, ScanRequest(pdg=True, material=True))
+    assert result.pdg is not None and result.material is not None
     return (
-        {v: i for i, v in enumerate(sorted(pdg_vals))},
-        {v: i for i, v in enumerate(sorted(mat_vals))},
+        {v: i for i, v in enumerate(sorted(result.pdg))},
+        {v: i for i, v in enumerate(sorted(result.material))},
     )
 
 
-def _accumulate_value_counts(counts: dict, series: pd.Series, cast) -> None:
-    for name, count in series.value_counts().items():
-        name = cast(name)
-        counts[name] = counts.get(name, 0) + int(count)
-
-
-def _rank_by_frequency_from_files(files: list[Path], column: str, cast) -> dict:
-    """Scan `column` across `files` and return `{cast(value): total_count}`,
-    accumulated in file order (see `fingerprint_files`'s docstring on why
-    scan order — not a normalized/sorted order — is preserved: it drives
-    tie-breaking in the frequency ranking below)."""
-    counts: dict = {}
-    for path in files:
-        df = pd.read_parquet(path, columns=[column])
-        _accumulate_value_counts(counts, df[column], cast)
-    return counts
-
-
-def _topn_plus_other_map(counts: dict, n_classes: int) -> tuple[dict, dict, dict]:
+def _topn_plus_other_map(counts: "Mapping[Any, ValueStat]", n_classes: int) -> tuple[dict, dict, dict]:
     """Frequency-capped value->index map: the `n_classes - 1` most frequent
     keys get their own index; every rarer key is bucketed into a shared
     "other" index (`n_classes - 1`).
 
+    `counts` maps each key to something with `.count` and `.first_seen`
+    attributes (`giant.data.scan.ValueStat`) — ties in `.count` are broken by
+    `.first_seen` (whichever value was scanned first: file order, then row
+    order within a file — see `giant.data.scan`'s module docstring). This is
+    an explicit, documented contract, not an accident of iteration order.
+
     Returns `(class_map, other_members, class_counts)` — `other_members` is
     `{key: count}` for every key bucketed into "other" (the empirical
     within-bucket distribution, for `other_policy = "sample"` at rollout);
@@ -270,15 +246,15 @@ def _topn_plus_other_map(counts: dict, n_classes: int) -> tuple[dict, dict, dict
     (gitea #44) needs and that would otherwise be dropped once `counts` is
     collapsed into `class_map`.
     """
-    ranked = sorted(counts, key=lambda k: counts[k], reverse=True)
+    ranked = sorted(counts, key=lambda k: (-counts[k].count, counts[k].first_seen))
     keep = ranked[: max(n_classes - 1, 0)]
     class_map = {k: i for i, k in enumerate(keep)}
-    class_counts = {i: counts[k] for i, k in enumerate(keep)}
+    class_counts = {i: counts[k].count for i, k in enumerate(keep)}
     other_idx = n_classes - 1
     other_members: dict = {}
     for k in ranked[len(keep) :]:
         class_map[k] = other_idx
-        other_members[k] = counts[k]
+        other_members[k] = counts[k].count
     if other_members:
         class_counts[other_idx] = sum(other_members.values())
     return class_map, other_members, class_counts
@@ -294,8 +270,11 @@ def build_process_map_from_files(files: list[Path], n_experts: int) -> dict[str,
     mirrors how `build_features` clamps the n_sec label to K_MAX for the
     fixed-width n_sec_head classifier.
     """
-    counts = _rank_by_frequency_from_files(files, "process", str)
-    class_map, _, _ = _topn_plus_other_map(counts, n_experts)
+    from giant.data.scan import ScanRequest, scan_metadata
+
+    result = scan_metadata(files, ScanRequest(process=True))
+    assert result.process is not None
+    class_map, _, _ = _topn_plus_other_map(result.process, n_experts)
     return class_map
 
 
@@ -327,8 +306,13 @@ def build_topn_map_from_files(files: list[Path], column: str, n_classes: int, ca
     later for `other_policy = "sample"` at rollout — computed now since it's
     free during this same scan.
     """
-    counts = _rank_by_frequency_from_files(files, column, cast)
-    class_map, other_members, class_counts = _topn_plus_other_map(counts, n_classes)
+    from giant.data.scan import ScanRequest, scan_metadata
+
+    if column != "material":
+        raise ValueError(f"build_topn_map_from_files only supports column='material', got {column!r}")
+    result = scan_metadata(files, ScanRequest(material=True))
+    assert result.material is not None
+    class_map, other_members, class_counts = _topn_plus_other_map(result.material, n_classes)
     return TopNMap(class_map=class_map, other_members=other_members, class_counts=class_counts)
 
 
@@ -349,16 +333,9 @@ def build_pdg_topn_map_from_files(files: list[Path], n_classes: int) -> TopNMap:
     join (see `_df_to_dict`'s `has_sec_lists` guard) — silently skipped for
     those, same convention as elsewhere in this module.
     """
-    counts: dict = {}
-    for path in files:
-        columns = ["pdg"]
-        has_sec = "sec_pdg_list" in pq.ParquetFile(path).schema_arrow.names
-        if has_sec:
-            columns.append("sec_pdg_list")
-        df = pd.read_parquet(path, columns=columns)
-        _accumulate_value_counts(counts, df["pdg"], int)
-        if has_sec:
-            exploded = df["sec_pdg_list"].explode().dropna()
-            _accumulate_value_counts(counts, exploded, int)
-    class_map, other_members, class_counts = _topn_plus_other_map(counts, n_classes)
+    from giant.data.scan import ScanRequest, scan_metadata
+
+    result = scan_metadata(files, ScanRequest(pooled_pdg=True))
+    assert result.pooled_pdg is not None
+    class_map, other_members, class_counts = _topn_plus_other_map(result.pooled_pdg, n_classes)
     return TopNMap(class_map=class_map, other_members=other_members, class_counts=class_counts)

giant/data/scan.py

@@ -0,0 +1,171 @@
+"""Fused metadata scan over one or more parquet files.
+
+`giant.pipeline.run_setup_stage` needs several distinct frequency summaries
+before training can start — the event-id → row-count index (for the train/val
+split), the pdg/material vocabularies, an optional physics-process count, and
+a pooled pdg count (primary + secondary species, for onehot conditioning).
+Each of those used to be its own full `pd.read_parquet(path, columns=[...])`
+per file (`giant.data.loader`'s old `_rank_by_frequency_from_files` /
+`build_index_maps_from_files` / `build_pdg_topn_map_from_files`) — up to five
+separate reads of the same file. `scan_metadata` answers all of them in one
+`pl.collect_all` per file instead, sharing the file open/decompress cost.
+
+Every requested count comes back keyed by value, as a `ValueStat(count,
+first_seen)`. `first_seen` is the value's row ordinal — file order (as given
+in `files`), then row order within a file — via `row_index_name` on the
+per-file lazy scan plus a running row offset across files. This is what
+`giant.data.loader._topn_plus_other_map`'s frequency-ranking tie-break keys
+on: among equally-frequent values, whichever was scanned first wins its own
+class slot. That is an explicit, documented contract (this module is where
+it's implemented), not an accident of iteration order.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+
+import numpy as np
+import polars as pl
+
+from giant.data.loader import _offset_event_id, event_id_offset
+
+
+@dataclass(frozen=True)
+class ScanRequest:
+    """Which aggregations to compute. Every field defaults off so a caller
+    only pays for what it actually needs."""
+
+    event_index: bool = False
+    pdg: bool = False
+    material: bool = False
+    process: bool = False
+    pooled_pdg: bool = False
+    """pdg ∪ exploded sec_pdg_list — both roles a PDG code plays (primary
+    species and secondary species), pooled into one count per code. See
+    `giant.data.loader.build_pdg_topn_map_from_files`'s docstring for why."""
+
+
+@dataclass(frozen=True)
+class ValueStat:
+    count: int
+    first_seen: int
+
+
+@dataclass
+class MetadataScan:
+    event_index: tuple[np.ndarray, np.ndarray] | None = None
+    """(unique_ids, counts), ids ascending — matches
+    `setup_cache.compute_event_index_from_files`'s return shape."""
+    pdg: dict[int, ValueStat] | None = None
+    material: dict[str, ValueStat] | None = None
+    process: dict[str, ValueStat] | None = None
+    pooled_pdg: dict[int, ValueStat] | None = None
+
+
+def _group_lazy(path: Path, column: str) -> pl.LazyFrame:
+    return (
+        pl.scan_parquet(path, row_index_name="__row")
+        .select(column, "__row")
+        .group_by(column)
+        .agg(pl.len().alias("__count"), pl.col("__row").min().alias("__first_row"))
+    )
+
+
+def _pooled_pdg_lazy(path: Path, has_sec_pdg_list: bool) -> pl.LazyFrame:
+    lf = pl.scan_parquet(path, row_index_name="__row")
+    parts = [lf.select(pl.col("pdg").alias("__val"), "__row")]
+    if has_sec_pdg_list:
+        parts.append(lf.select(pl.col("sec_pdg_list").alias("__val"), "__row").explode("__val").drop_nulls("__val"))
+    combined = pl.concat(parts)
+    return combined.group_by("__val").agg(pl.len().alias("__count"), pl.col("__row").min().alias("__first_row"))
+
+
+def _merge_counts(acc: dict, df: pl.DataFrame, column: str, row_offset: int, cast) -> None:
+    for key, count, first_row in zip(
+        df[column].to_list(), df["__count"].to_list(), df["__first_row"].to_list(), strict=True
+    ):
+        key = cast(key)
+        first_seen = row_offset + int(first_row)
+        if key in acc:
+            prev_count, prev_first = acc[key]
+            acc[key] = (prev_count + int(count), min(prev_first, first_seen))
+        else:
+            acc[key] = (int(count), first_seen)
+
+
+def scan_metadata(files: list[Path], request: ScanRequest) -> MetadataScan:
+    """Scan `files` once (one `pl.collect_all` per file) and return every
+    aggregation `request` asks for. Files with zero rows contribute nothing
+    but still advance nothing (no row_offset change, nothing to merge)."""
+    event_id_parts: list[tuple[np.ndarray, np.ndarray]] = []
+    pdg_acc: dict[int, tuple[int, int]] = {}
+    material_acc: dict[str, tuple[int, int]] = {}
+    process_acc: dict[str, tuple[int, int]] = {}
+    pooled_pdg_acc: dict[int, tuple[int, int]] = {}
+
+    row_offset = 0
+    for file_idx, path in enumerate(files):
+        keys: list[str] = []
+        lazies: list[pl.LazyFrame] = []
+
+        if request.event_index:
+            keys.append("event_id")
+            lazies.append(_group_lazy(path, "event_id"))
+        if request.pdg:
+            keys.append("pdg")
+            lazies.append(_group_lazy(path, "pdg"))
+        if request.material:
+            keys.append("material")
+            lazies.append(_group_lazy(path, "material"))
+        if request.process:
+            keys.append("process")
+            lazies.append(_group_lazy(path, "process"))
+        if request.pooled_pdg:
+            has_sec = "sec_pdg_list" in pl.scan_parquet(path).collect_schema().names()
+            keys.append("pooled_pdg")
+            lazies.append(_pooled_pdg_lazy(path, has_sec))
+
+        keys.append("__n")
+        lazies.append(pl.scan_parquet(path).select(pl.len().alias("__n")))
+
+        results = dict(zip(keys, pl.collect_all(lazies, engine="streaming"), strict=True))
+        n_rows = int(results["__n"].item()) if len(results["__n"]) else 0
+
+        if request.event_index:
+            df = results["event_id"]
+            ids = _offset_event_id(df["event_id"].to_numpy(), event_id_offset(file_idx))
+            counts = df["__count"].to_numpy().astype(np.int64)
+            if ids.size:
+                event_id_parts.append((ids, counts))
+        if request.pdg:
+            _merge_counts(pdg_acc, results["pdg"], "pdg", row_offset, int)
+        if request.material:
+            _merge_counts(material_acc, results["material"], "material", row_offset, str)
+        if request.process:
+            _merge_counts(process_acc, results["process"], "process", row_offset, str)
+        if request.pooled_pdg:
+            _merge_counts(pooled_pdg_acc, results["pooled_pdg"], "__val", row_offset, int)
+
+        row_offset += n_rows
+
+    event_index = None
+    if request.event_index:
+        if event_id_parts:
+            all_ids = np.concatenate([p[0] for p in event_id_parts])
+            all_counts = np.concatenate([p[1] for p in event_id_parts])
+            order = np.argsort(all_ids, kind="stable")
+            event_index = (all_ids[order], all_counts[order])
+        else:
+            event_index = (np.empty(0, dtype=np.int64), np.empty(0, dtype=np.int64))
+
+    def _to_stats(acc: dict) -> dict:
+        return {k: ValueStat(*v) for k, v in acc.items()}
+
+    return MetadataScan(
+        event_index=event_index,
+        pdg=_to_stats(pdg_acc) if request.pdg else None,
+        material=_to_stats(material_acc) if request.material else None,
+        process=_to_stats(process_acc) if request.process else None,
+        pooled_pdg=_to_stats(pooled_pdg_acc) if request.pooled_pdg else None,
+    )

giant/data/setup_cache.py

@@ -23,7 +23,7 @@ import numpy as np
 
 from giant import config
 from giant.constants import COND_DIM, K_MAX, PARTICLE_PHYS_DIM, SEC_SLOT_DIM, X_DIM
-from giant.data.loader import TopNMap, event_id_offset, load_event_ids
+from giant.data.loader import TopNMap
 from giant.data.transforms import Normalizer, sorted_membership
 
 # Bump manually on a change to the data-encoding semantics (e.g. a future
@@ -349,12 +349,21 @@ def save(
 
 
 def compute_event_index_from_files(files: list[Path]) -> tuple[np.ndarray, np.ndarray]:
-    """Unique event ids + per-event row (step) counts, across all `files`."""
+    """Unique event ids + per-event row (step) counts, across all `files`.
+
+    Computed via a streaming per-file `group_by("event_id")` (see
+    `giant.data.scan.scan_metadata`) rather than concatenating every row's
+    raw event_id across every file before `np.unique` — the latter's peak
+    memory is 8 bytes x total row count; this is bounded by the (much
+    smaller) unique event count instead.
+    """
+    from giant.data.scan import ScanRequest, scan_metadata
+
     if not files:
         return np.empty(0, dtype=np.int64), np.empty(0, dtype=np.int64)
-    all_ids = np.concatenate([load_event_ids(f, offset=event_id_offset(i)) for i, f in enumerate(files)])
-    unique_ids, counts = np.unique(all_ids, return_counts=True)
-    return unique_ids, counts
+    result = scan_metadata(files, ScanRequest(event_index=True))
+    assert result.event_index is not None
+    return result.event_index
 
 
 def n_train_steps_for_split(unique_ids: np.ndarray, counts: np.ndarray, train_events_arr: np.ndarray) -> int:

giant/geometry.py

@@ -28,7 +28,7 @@ from pathlib import Path
 from typing import Any, Iterable
 
 import numpy as np
-import pandas as pd
+import polars as pl
 import pyarrow.parquet as pq
 
 _INSTALL_HINT = "the geometry oracle needs scikit-learn — install it with `uv sync --extra cpu --extra geometry`"
@@ -301,14 +301,18 @@ def _fit_slab_lookup(
     edges = np.linspace(z_min, z_max, n_bins + 1)
     bin_idx = np.clip(np.searchsorted(edges, z, side="right") - 1, 0, n_bins - 1)
 
+    # pandas' groupby(...).size() sorts group keys ascending by default, so
+    # a tie in `n` for the same bin (equal counts split between two
+    # material/layer_id combos) resolves to the lexicographically-first
+    # combo — matched here by sorting on the keys first, then a
+    # maintain_order-stable sort on `n` so ties keep that key order.
     counts = (
-        pd.DataFrame({"bin": bin_idx, "material": mat, "layer_id": lay})
-        .groupby(["bin", "material", "layer_id"])
-        .size()
-        .to_frame("n")
-        .reset_index()
-        .sort_values("n", ascending=False)
-        .drop_duplicates("bin")
+        pl.DataFrame({"bin": bin_idx, "material": mat, "layer_id": lay})
+        .group_by(["bin", "material", "layer_id"])
+        .agg(pl.len().alias("n"))
+        .sort(["bin", "material", "layer_id"])
+        .sort("n", descending=True, maintain_order=True)
+        .unique(subset="bin", keep="first", maintain_order=True)
     )
 
     bin_material = np.full(n_bins, "", dtype=object)

giant/pipeline.py

@@ -15,14 +15,12 @@ from giant.constants import (
 from giant.data import setup_cache
 from giant.data.loader import (
     TopNMap,
+    _topn_plus_other_map,
     event_id_offset,
     find_parquet_files,
     iter_file_chunks,
-    build_index_maps_from_files,
-    build_pdg_topn_map_from_files,
-    build_process_map_from_files,
-    build_topn_map_from_files,
 )
+from giant.data.scan import MetadataScan, ScanRequest, scan_metadata
 from giant.data.transforms import (
     Normalizer,
     build_features,
@@ -127,12 +125,71 @@ def run_setup_stage(
             loaded = setup_cache.load(data, files, echo=echo)
         cache = loaded if loaded is not None else setup_cache.SetupCache.empty(files)
 
-    if cache is not None and cache.event_index is not None:
+    # Every section below first asks the cache; whatever's missing is
+    # collected into one ScanRequest and answered by a single fused scan
+    # (giant.data.scan.scan_metadata), instead of a separate full pass per
+    # section (event index, vocab, process counts, pdg/material top-N counts
+    # used to each re-open and re-read every file on their own).
+    particle_cfg = cfg["conditioning"]["particle"]
+    material_cfg = cfg["conditioning"]["material"]
+    particle_type_cfg = config.ParticleTypeConfig.from_dict(cfg["stage2_model"].get("particle_type"))
+    particle_type_target = particle_type_cfg.target
+
+    # A process map is needed if either stage's router reads the physics
+    # process label (type="process"). Only one map is built even if both
+    # stages want one — see the module-level note in giant/cli.py's
+    # _router_total_experts for why composed-router n_experts isn't a plain
+    # int; process routers are never composed in practice, so this doesn't
+    # need that generality.
+    process_router_cfg = next(
+        (r for r in (stage1_router, stage2_router) if r.get("enabled") and r.get("type") == "process"),
+        None,
+    )
+    process_n_experts = process_router_cfg["n_experts"] if process_router_cfg is not None else None
+
+    need_pdg_onehot = particle_cfg["type"] == "onehot"
+    need_sec_type_onehot = particle_type_target == "onehot"
+    sec_type_n_classes = (
+        resolve_type_n_classes(particle_type_cfg, particle_cfg["emb_dim"]) if need_sec_type_onehot else None
+    )
+    need_material_onehot = material_cfg["type"] == "onehot"
+    material_n_classes = material_cfg["emb_dim"] if need_material_onehot else None
+
+    def _topn_cached(axis: str, n_classes: int) -> TopNMap | None:
+        return cache.topn_maps.get(setup_cache.topn_key(axis, n_classes)) if cache is not None else None
+
+    need_event_index = cache is None or cache.event_index is None
+    need_vocab = cache is None or cache.vocab is None
+    need_process = process_n_experts is not None and (cache is None or cache.proc_maps.get(process_n_experts) is None)
+    # The PDG axis is used independently by conditioning.particle.type="onehot"
+    # (cond_cat's onehot feature) and stage2_model.particle_type.target="onehot"
+    # (secondary-species decode) — their class counts can now differ (gitea
+    # #29: stage2_model.particle_type.n_classes, 0 = inherit
+    # conditioning.particle.emb_dim), but both are built from the same
+    # pooled pdg-count scan, so a cache miss on either one asks for it.
+    need_pdg_pooled = (need_pdg_onehot and _topn_cached("pdg", particle_cfg["emb_dim"]) is None) or (
+        need_sec_type_onehot and sec_type_n_classes is not None and _topn_cached("pdg", sec_type_n_classes) is None
+    )
+    need_material_topn = (
+        need_material_onehot and material_n_classes is not None and _topn_cached("material", material_n_classes) is None
+    )
+
+    request = ScanRequest(
+        event_index=need_event_index,
+        pdg=need_vocab,
+        material=need_vocab or need_material_topn,
+        process=need_process,
+        pooled_pdg=need_pdg_pooled,
+    )
+    scan = scan_metadata(files, request) if request != ScanRequest() else MetadataScan()
+
+    if not need_event_index:
         unique_ids, counts = cache.event_index
         echo(f"event index: cache hit ({len(unique_ids):,} unique events)")
     else:
         echo("scanning event IDs …")
-        unique_ids, counts = setup_cache.compute_event_index_from_files(files)
+        assert scan.event_index is not None
+        unique_ids, counts = scan.event_index
         if cache is not None:
             cache.event_index = (unique_ids, counts)
 
@@ -141,55 +198,34 @@ def run_setup_stage(
     n_train_steps = setup_cache.n_train_steps_for_split(unique_ids, counts, events_arr)
     echo(f"  {int(counts.sum()):,} steps  |  {len(train_events)} train events  |  {len(val_events)} val events")
 
-    if cache is not None and cache.vocab is not None:
+    if not need_vocab:
         pdg_map, mat_map = cache.vocab
         echo(f"vocabulary maps: cache hit ({len(pdg_map)} PDG codes, {len(mat_map)} materials)")
     else:
         echo("building vocabulary maps …")
-        pdg_map, mat_map = build_index_maps_from_files(files)
+        assert scan.pdg is not None and scan.material is not None
+        pdg_map = {v: i for i, v in enumerate(sorted(scan.pdg))}
+        mat_map = {v: i for i, v in enumerate(sorted(scan.material))}
         echo(f"  {len(pdg_map)} PDG codes  |  {len(mat_map)} materials")
         if cache is not None:
             cache.vocab = (pdg_map, mat_map)
 
-    # A process map is needed if either stage's router reads the physics
-    # process label (type="process"). Only one map is built even if both
-    # stages want one — see the module-level note in giant/cli.py's
-    # _router_total_experts for why composed-router n_experts isn't a plain
-    # int; process routers are never composed in practice, so this doesn't
-    # need that generality.
     proc_map: dict[str, int] | None = None
-    process_router_cfg = next(
-        (r for r in (stage1_router, stage2_router) if r.get("enabled") and r.get("type") == "process"),
-        None,
-    )
     if process_router_cfg is not None:
-        n_experts = process_router_cfg["n_experts"]
-        cached_proc_map = cache.proc_maps.get(n_experts) if cache is not None else None
-        if cached_proc_map is not None:
+        assert process_n_experts is not None
+        if not need_process:
+            assert cache is not None
+            cached_proc_map = cache.proc_maps.get(process_n_experts)
+            assert cached_proc_map is not None
             proc_map = cached_proc_map
-            echo(f"process vocabulary: cache hit ({len(proc_map)} labels, {n_experts} experts)")
+            echo(f"process vocabulary: cache hit ({len(proc_map)} labels, {process_n_experts} experts)")
         else:
             echo("building process vocabulary …")
-            proc_map = build_process_map_from_files(files, n_experts=n_experts)
-            echo(f"  {len(proc_map)} process labels mapped to {n_experts} experts")
+            assert scan.process is not None
+            proc_map, _, _ = _topn_plus_other_map(scan.process, process_n_experts)
+            echo(f"  {len(proc_map)} process labels mapped to {process_n_experts} experts")
             if cache is not None:
-                cache.proc_maps[n_experts] = proc_map
-
-    # Top-N-plus-other maps for onehot conditioning/type axes.
-    # The PDG axis is used independently by conditioning.particle.type="onehot"
-    # (cond_cat's onehot feature) and stage2_model.particle_type.target="onehot"
-    # (secondary-species decode) — their class counts can now differ (gitea
-    # #29: stage2_model.particle_type.n_classes, 0 = inherit
-    # conditioning.particle.emb_dim), so each is resolved and built
-    # independently via _pdg_topn below. cache.topn_maps is keyed by
-    # (axis, n_classes) (setup_cache.topn_key), so when the two resolve to
-    # the same N the second call is a cache hit against the first — no extra
-    # scan in the common case where they still match. The material axis is
-    # independent of both.
-    particle_cfg = cfg["conditioning"]["particle"]
-    material_cfg = cfg["conditioning"]["material"]
-    particle_type_cfg = config.ParticleTypeConfig.from_dict(cfg["stage2_model"].get("particle_type"))
-    particle_type_target = particle_type_cfg.target
+                cache.proc_maps[process_n_experts] = proc_map
 
     def _pdg_topn(n_classes: int) -> TopNMap:
         cache_key = setup_cache.topn_key("pdg", n_classes)
@@ -198,33 +234,36 @@ def run_setup_stage(
             echo(f"pdg top-N map: cache hit ({len(cached.class_map)} codes, {n_classes} classes)")
             return cached
         echo("building pdg top-N map …")
-        topn_map = build_pdg_topn_map_from_files(files, n_classes=n_classes)
+        assert scan.pooled_pdg is not None
+        class_map, other_members, class_counts = _topn_plus_other_map(scan.pooled_pdg, n_classes)
+        topn_map = TopNMap(class_map=class_map, other_members=other_members, class_counts=class_counts)
         echo(f"  {len(topn_map.class_map)} pdg codes mapped to {n_classes} classes")
         if cache is not None:
             cache.topn_maps[cache_key] = topn_map
         return topn_map
 
-    pdg_topn_map: TopNMap | None = None
-    if particle_cfg["type"] == "onehot":
-        pdg_topn_map = _pdg_topn(particle_cfg["emb_dim"])
-
+    pdg_topn_map: TopNMap | None = _pdg_topn(particle_cfg["emb_dim"]) if need_pdg_onehot else None
     sec_type_topn_map: TopNMap | None = None
-    if particle_type_target == "onehot":
-        sec_type_n_classes = resolve_type_n_classes(particle_type_cfg, particle_cfg["emb_dim"])
+    if need_sec_type_onehot:
+        assert sec_type_n_classes is not None
         sec_type_topn_map = _pdg_topn(sec_type_n_classes)
 
     mat_topn_map: TopNMap | None = None
-    if material_cfg["type"] == "onehot":
-        n_classes = material_cfg["emb_dim"]
-        cache_key = setup_cache.topn_key("material", n_classes)
+    if need_material_onehot:
+        assert material_n_classes is not None
+        cache_key = setup_cache.topn_key("material", material_n_classes)
         cached = cache.topn_maps.get(cache_key) if cache is not None else None
         if cached is not None:
             mat_topn_map = cached
-            echo(f"material top-N map: cache hit ({len(mat_topn_map.class_map)} materials, {n_classes} classes)")
+            echo(
+                f"material top-N map: cache hit ({len(mat_topn_map.class_map)} materials, {material_n_classes} classes)"
+            )
         else:
             echo("building material top-N map …")
-            mat_topn_map = build_topn_map_from_files(files, "material", n_classes=n_classes, cast=str)
-            echo(f"  {len(mat_topn_map.class_map)} materials mapped to {n_classes} classes")
+            assert scan.material is not None
+            class_map, other_members, class_counts = _topn_plus_other_map(scan.material, material_n_classes)
+            mat_topn_map = TopNMap(class_map=class_map, other_members=other_members, class_counts=class_counts)
+            echo(f"  {len(mat_topn_map.class_map)} materials mapped to {material_n_classes} classes")
             if cache is not None:
                 cache.topn_maps[cache_key] = mat_topn_map
 
@@ -435,7 +474,6 @@ def run_train_job(
         mat_topn_map=cond_mat_topn,
         sec_type_class_map=sec_type_class_map,
         k_max=k_max,
-        seed=t["seed"],
     )
     val_ds = StreamingStepsDataset(
         files=files,
@@ -457,17 +495,30 @@ def run_train_job(
     )
 
     pin = device.type == "cuda"
+    # DataLoader worker subprocesses default to fork() on Linux, but by the
+    # time they're created this process has already run polars queries
+    # (run_setup_stage's fused metadata scan, above) — polars' native
+    # (rayon) thread pool doesn't survive a fork: a worker that inherits it
+    # mid-fork deadlocks the instant it touches polars itself, which
+    # StreamingStepsDataset's iter_file_chunks now does on every row group.
+    # "spawn" starts each worker as a fresh interpreter with no inherited
+    # thread-pool state, avoiding that hazard entirely. Only matters when
+    # workers actually exist — num_workers=0 runs the dataset in-process and
+    # never forks.
+    mp_context = "spawn" if num_workers > 0 else None
     train_loader = DataLoader(
         train_ds,
         batch_size=None,
         num_workers=num_workers,
         pin_memory=pin,
+        multiprocessing_context=mp_context,
     )
     val_loader = DataLoader(
         val_ds,
         batch_size=None,
         num_workers=num_workers,
         pin_memory=pin,
+        multiprocessing_context=mp_context,
     )
 
     model_config = {

giant/tools/profile_analysis_costs.py

@@ -26,7 +26,7 @@ import numpy as np
 import polars as pl
 
 from giant.analysis.catalog import catalog_ids, get_spec
-from giant.analysis.run import compute_reduced
+from giant.analysis.condor import compute_reduced
 from giant.analysis.context import build_context
 from giant.analysis.sources import RolloutSpec
 

giant/tools/profile_setup_scan.py

@@ -0,0 +1,162 @@
+"""Benchmark `giant.pipeline.run_setup_stage`'s cold-cache scan against synthetic data.
+
+Generates a schema-complete synthetic steps parquet (matching
+`tests/test_pipeline.py`'s `_make_synthetic_steps`, but built with vectorized
+numpy instead of a per-row Python loop so it scales to millions of rows) at a
+few row counts, times `run_setup_stage` with `cache_setup=False` (so every
+call is a genuine cold scan, never served from the sidecar), and prints a
+before/after-style table. Run this on `master` before a change and again
+after to see what a step actually bought — see the "speed up dwarf
+warm-cache" plan for the pass-by-pass breakdown this benchmark is meant to
+attribute (giant/data/loader.py, giant/data/scan.py, giant/pipeline.py).
+
+Usage: ``uv run python giant/tools/profile_setup_scan.py``
+"""
+
+from __future__ import annotations
+
+import time
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+import numpy as np
+import polars as pl
+
+from giant import config as gconfig
+from giant.pipeline import run_setup_stage
+
+ROW_COUNTS = [20_000, 100_000, 500_000, 2_000_000]
+
+_MATERIALS = ["G4_AIR", "G4_Fe"]
+_PDGS = [11, 22]
+_PROCESSES = ["eIoni", "phot", "compt"]
+
+
+def _unit_vectors(n: int, rng: np.random.Generator) -> np.ndarray:
+    v = rng.normal(size=(n, 3))
+    return v / np.linalg.norm(v, axis=1, keepdims=True)
+
+
+def _ragged_lists(k: np.ndarray, rng: np.random.Generator, lo: float, hi: float) -> list[list[float]]:
+    total = int(k.sum())
+    flat = rng.uniform(lo, hi, size=total)
+    idx = np.cumsum(k)[:-1]
+    return [arr.tolist() for arr in np.split(flat, idx)]
+
+
+def _make_synthetic_steps(n: int, seed: int = 0) -> pl.DataFrame:
+    """Vectorized equivalent of tests/test_pipeline.py's `_make_synthetic_steps`.
+
+    event_id is assigned so each event gets 2-3 steps (matching that
+    fixture's structure), and pdg/material/process cycle deterministically
+    by row index rather than being drawn at random, same as the original.
+    """
+    rng = np.random.default_rng(seed)
+    n_events = max(n // 3, 1)
+
+    pre_E = rng.uniform(50.0, 500.0, size=n)
+    n_sec = rng.integers(0, 3, size=n)
+    frac_dep = rng.uniform(0.05, 0.3, size=n)
+    frac_sec = np.where(n_sec > 0, rng.uniform(0.05, 0.2, size=n), 0.0)
+    frac_post = 1.0 - frac_dep - frac_sec
+    edep = pre_E * frac_dep
+    e_sec = pre_E * frac_sec
+    post_E = pre_E * frac_post
+    pre_pos = rng.uniform(-10, 10, size=(n, 3))
+    step_length = rng.uniform(0.1, 5.0, size=n)
+    pre_dir = np.zeros((n, 3))
+    pre_dir[:, 2] = 1.0
+    post_dir = _unit_vectors(n, rng)
+    post_pos = pre_pos + step_length[:, None] * pre_dir
+
+    row_idx = np.arange(n)
+    event_id = row_idx % n_events
+
+    sec_E = _ragged_lists(n_sec, rng, 0.1, 1.0)  # placeholder magnitude, rescaled below
+    sec_dx = _ragged_lists(n_sec, rng, -1.0, 1.0)
+    sec_dy = _ragged_lists(n_sec, rng, -1.0, 1.0)
+    sec_dz = _ragged_lists(n_sec, rng, -1.0, 1.0)
+    total_sec = int(n_sec.sum())
+    flat_pdg = [_PDGS[(row_idx[i] + j) % 2] for i in range(n) for j in range(n_sec[i])]
+    idx = np.cumsum(n_sec)[:-1]
+    sec_pdg = (
+        [list(x) for x in np.split(np.array(flat_pdg, dtype=np.int64), idx)] if total_sec else [[] for _ in range(n)]
+    )
+    # Rescale each row's secondary energies to sum to that row's e_sec (a
+    # Dirichlet split, like the original fixture) rather than the raw
+    # uniform placeholder.
+    sec_E_scaled = []
+    for i in range(n):
+        vals = np.array(sec_E[i])
+        if vals.size:
+            sec_E_scaled.append((vals / vals.sum() * e_sec[i]).tolist())
+        else:
+            sec_E_scaled.append([])
+
+    return pl.DataFrame(
+        {
+            "event_id": event_id,
+            "pdg": np.array(_PDGS)[row_idx % 2],
+            "pre_x": pre_pos[:, 0],
+            "pre_y": pre_pos[:, 1],
+            "pre_z": pre_pos[:, 2],
+            "pre_E": pre_E,
+            "pre_dx": pre_dir[:, 0],
+            "pre_dy": pre_dir[:, 1],
+            "pre_dz": pre_dir[:, 2],
+            "material": np.array(_MATERIALS)[row_idx % 2],
+            "layer_id": row_idx % 5,
+            "child_track_ids": [list(range(int(k))) for k in n_sec],
+            "e_sec": e_sec,
+            "process": np.array(_PROCESSES)[row_idx % 3],
+            "step_length": step_length,
+            "post_E": post_E,
+            "edep": edep,
+            "post_dx": post_dir[:, 0],
+            "post_dy": post_dir[:, 1],
+            "post_dz": post_dir[:, 2],
+            "post_x": post_pos[:, 0],
+            "post_y": post_pos[:, 1],
+            "post_z": post_pos[:, 2],
+            "sec_E_list": sec_E_scaled,
+            "sec_pdg_list": sec_pdg,
+            "sec_dx_list": sec_dx,
+            "sec_dy_list": sec_dy,
+            "sec_dz_list": sec_dz,
+        }
+    )
+
+
+def _time_setup_stage(data: Path) -> float:
+    cfg = gconfig.merge_cli_overrides(gconfig.DEFAULT_CONFIG, None, {})
+    gconfig.validate_config(cfg)
+    t0 = time.perf_counter()
+    run_setup_stage(
+        data,
+        val_fraction=cfg["train"]["val_fraction"],
+        seed=cfg["train"]["seed"],
+        cfg=cfg,
+        cache_setup=False,
+        echo=lambda *a, **k: None,
+    )
+    return time.perf_counter() - t0
+
+
+def main() -> None:
+    with TemporaryDirectory(prefix="giant-setup-scan-profile-") as tmp:
+        tmp_path = Path(tmp)
+        print(f"{'n_rows':>10s}  {'time (s)':>10s}  {'rows/s':>12s}")
+        for n in ROW_COUNTS:
+            path = tmp_path / f"steps_{n}.parquet"
+            _make_synthetic_steps(n).write_parquet(path)
+            # warm the OS page cache so the timed pass measures compute, not
+            # the one-time cold read of a freshly-written file.
+            pl.scan_parquet(path).select(pl.len()).collect()
+
+            dt = _time_setup_stage(path)
+            print(f"{n:>10,d}  {dt:>10.3f}  {n / dt:>12,.0f}")
+            path.unlink()
+
+
+if __name__ == "__main__":
+    main()

giant/training/loop.py

@@ -18,7 +18,6 @@ import torch
 from torch.utils.data import DataLoader
 from tqdm import tqdm
 
-from giant import config
 from giant.data.loader import TopNMap
 from giant.data.setup_cache import topnmap_to_json
 from giant.training.checkpoint import build_checkpoint, init_stages_from_checkpoints, load_checkpoint
@@ -185,17 +184,6 @@ def train(
             if device.type == "cuda":
                 torch.cuda.reset_peak_memory_stats(device)
             collector.start_epoch(epoch)
-            # Epoch-aware RNG: same noise (and, below, same batch order) for
-            # epoch k whether the run is one process or a chain of per-epoch
-            # jobs. See giant.config.epoch_seed.
-            config.seed_everything(config.epoch_seed(t["seed"], epoch))
-            # Epoch-aware shuffle stream (see StreamingStepsDataset.set_epoch):
-            # keeps epoch k's batch order identical whether it runs here or as
-            # its own resumed per-epoch job in a b2luigi workflow.
-            # (tests hand `train` a plain list of batches, which has neither)
-            set_epoch = getattr(getattr(train_loader, "dataset", None), "set_epoch", None)
-            if callable(set_epoch):
-                set_epoch(epoch)
             for trainer in trainers.values():
                 trainer.train_mode()
 

giant/training/plots.py

@@ -115,7 +115,7 @@ def derive_metrics_dir(
 
     Precedence: an explicit `out_dir` always wins. Otherwise
     `default_base / f"metrics_{run_dir.name}"` (the CLI passes the repo's
-    gitignored `analysis_runs/`, matching `giant.analysis.run.derive_run_dir`'s
+    gitignored `analysis_runs/`, matching `giant.analysis.condor.derive_run_dir`'s
     convention) — training-progress plots live alongside rollout-vs-reference
     analysis runs, not inside the training run directory itself.
     """

giant/workflow/__init__.py

@@ -1,40 +0,0 @@
-"""b2luigi orchestration of the full GIANT pipeline.
-
-One workflow TOML (``spec.py``) parameterises an entire experiment — dataset,
-geometry oracle, N trainings, N rollouts, N analyses — and ``giant workflow
-run <spec.toml>`` turns it into a b2luigi task graph (``tasks.py``) whose
-targets are files on ``/ceph``: nothing is recomputed that already exists,
-every step waits for its inputs, and HTCondor submission/polling is b2luigi's
-job rather than a hand-rolled submit-file generator.
-
-This is the only sanctioned way to run a multi-step pipeline; ``giant`` and
-``dwarf`` stay single-step primitives that these tasks invoke.
-
-``tasks``/``run`` import b2luigi, so they are *not* imported here — a plain
-``import giant.workflow`` (or ``giant.workflow.spec``) works without the
-``workflow`` extra installed.
-"""
-
-from giant.workflow.spec import (
-    AnalysisSpec,
-    CondorSpec,
-    DatasetSpec,
-    GeometrySpec,
-    RolloutSpec,
-    TrainSpec,
-    WorkflowSpec,
-    load_spec,
-    spec_hash,
-)
-
-__all__ = [
-    "AnalysisSpec",
-    "CondorSpec",
-    "DatasetSpec",
-    "GeometrySpec",
-    "RolloutSpec",
-    "TrainSpec",
-    "WorkflowSpec",
-    "load_spec",
-    "spec_hash",
-]

giant/workflow/htcondor.py

@@ -1,82 +0,0 @@
-"""HTCondor job descriptions for the workflow tasks.
-
-b2luigi writes every key of a task's ``htcondor_settings`` dict straight into
-that job's submit description, so these helpers are just the ETP-specific
-resource/requirement conventions in one place:
-
-* **CPU jobs** (setup cache, geometry oracle, analysis compute) keep what
-  the deleted ``giant analyze submit`` used: ``+RemoteJob`` for grid I/O, or
-  ``TARGET.ProvidesETPResources`` when the files are local to the cluster.
-* **GPU jobs** (training epochs, rollout) are remote-only, so they always
-  carry ``+RemoteJob`` and reach ``/ceph`` through
-  ``TARGET.ProvidesEtpCeph`` — the requirement strings are ported from the
-  ``condor-gpu-train-rollout`` branch's ``giant/condor.py`` rather than
-  rewritten, since they encode what the ETP HTCondor wiki documents for
-  TOpAS/NEMO2 GPU workers.
-"""
-
-from __future__ import annotations
-
-from giant.workflow.spec import CondorSpec
-
-__all__ = ["cpu_settings", "gpu_settings", "gpu_requirements"]
-
-
-def cpu_settings(
-    condor: CondorSpec,
-    *,
-    request_memory_mb: int | None = None,
-    request_cpus: int | None = None,
-    walltime_s: int | None = None,
-) -> dict:
-    settings: dict = {
-        "universe": "docker",
-        "docker_image": condor.docker_image_cpu,
-        "request_memory": request_memory_mb if request_memory_mb is not None else condor.request_memory_mb,
-        "request_cpus": request_cpus if request_cpus is not None else condor.request_cpus,
-        "accounting_group": condor.accounting_group,
-        "should_transfer_files": "YES",
-        "when_to_transfer_output": "ON_EXIT",
-    }
-    if condor.remote:
-        settings["+RemoteJob"] = "True"
-    else:
-        settings["requirements"] = "TARGET.ProvidesETPResources"
-    if walltime_s is not None:
-        settings["+RequestWalltime"] = int(walltime_s)
-    return settings
-
-
-def gpu_requirements(gpu_type: str | None = None, gpu_memory_mb: int | None = None) -> str:
-    """``TARGET.ProvidesEtpCeph`` (remote /ceph access) ANDed with any GPU pin."""
-    clauses = ["TARGET.ProvidesEtpCeph =?= True"]
-    if gpu_type is not None:
-        clauses.append(f'TARGET.GPUs_DeviceName =?= "{gpu_type}"')
-    if gpu_memory_mb is not None:
-        clauses.append(f"TARGET.GPUs_GlobalMemoryMb >= {gpu_memory_mb}")
-    return " && ".join(clauses)
-
-
-def gpu_settings(
-    condor: CondorSpec,
-    *,
-    request_gpus: int = 1,
-    gpu_type: str | None = None,
-    gpu_memory_mb: int | None = None,
-    request_memory_mb: int = 16384,
-    request_cpus: int = 4,
-    walltime_s: int = 86400,
-) -> dict:
-    return {
-        "universe": "docker",
-        "docker_image": condor.docker_image_gpu,
-        "request_memory": request_memory_mb,
-        "request_cpus": request_cpus,
-        "RequestGPUs": request_gpus,
-        "+RequestWalltime": int(walltime_s),
-        "accounting_group": condor.accounting_group,
-        "should_transfer_files": "YES",
-        "when_to_transfer_output": "ON_EXIT",
-        "+RemoteJob": "True",
-        "requirements": f"({gpu_requirements(gpu_type, gpu_memory_mb)})",
-    }

giant/workflow/run.py

@@ -1,103 +0,0 @@
-#!/usr/bin/env python
-"""Entry point b2luigi re-executes on every worker.
-
-Locally this is what ``giant workflow run <spec.toml>`` execs; on a batch
-worker it is what the generated wrapper script runs (after ``cd repo_dir`` and
-sourcing ``env_script``), with ``--spec`` forwarded via the
-``task_cmd_additional_args`` setting so the worker resolves exactly the same
-spec — and therefore the same task graph and output paths — as the submitter.
-
-b2luigi needs a real script path for that re-execution, which is why this is a
-script rather than a ``python -m`` module.
-"""
-
-from __future__ import annotations
-
-import argparse
-import sys
-from pathlib import Path
-
-# Allow `python giant/workflow/run.py` from a checkout that isn't installed.
-sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
-
-import b2luigi  # noqa: E402
-
-from giant.workflow.spec import WorkflowSpec, load_spec  # noqa: E402
-from giant.workflow.tasks import WorkflowTask, set_spec  # noqa: E402
-
-
-def build_parser() -> argparse.ArgumentParser:
-    parser = argparse.ArgumentParser(description="Run a GIANT workflow spec with b2luigi.")
-    parser.add_argument("--spec", required=True, help="Workflow TOML (see configs/workflow_example.toml)")
-    parser.add_argument("--workers", type=int, default=1, help="Concurrent luigi workers")
-    parser.add_argument(
-        "--batch",
-        action="store_true",
-        help="Submit batch-system tasks to HTCondor (otherwise everything runs locally)",
-    )
-    parser.add_argument(
-        "--mode",
-        choices=("run", "dry-run", "show-output", "remove"),
-        default="run",
-        help="run (default), dry-run (print pending tasks), show-output (print every target), remove (delete outputs)",
-    )
-    parser.add_argument("--scheduler-host", default=None, help="luigid host (default: local scheduler)")
-    parser.add_argument("--scheduler-port", type=int, default=None, help="luigid port")
-    return parser
-
-
-def configure(spec: WorkflowSpec, spec_path: Path, batch: bool) -> None:
-    """Wire b2luigi's settings from the spec.
-
-    ``/ceph`` is shared between submit host and workers, so there is
-    deliberately no ``transfer_files``: ``result_dir``/``log_dir`` must live
-    somewhere both sides can see.
-    """
-    set_spec(spec)
-
-    b2luigi.set_setting("result_dir", spec.result_dir)
-    b2luigi.set_setting("log_dir", spec.log_dir)
-    b2luigi.set_setting("task_file_dir", str(Path(spec.result_dir) / "task_files"))
-    b2luigi.set_setting("use_parameter_name_in_output", True)
-    b2luigi.set_setting("batch_system", "htcondor" if batch else "local")
-    b2luigi.set_setting("working_dir", spec.condor.repo_dir)
-    b2luigi.set_setting("job_name", spec.name)
-    if spec.condor.env_script:
-        b2luigi.set_setting("env_script", spec.condor.env_script)
-    # The worker command is `<executable> [<basename of this file>] --batch-runner
-    # --task-id ...`, run after `cd working_dir`. Only the *basename* would be
-    # used, so the filename is dropped and the repo-relative script path is
-    # made part of the executable instead.
-    b2luigi.set_setting("add_filename_to_cmd", False)
-    b2luigi.set_setting("executable", [".venv/bin/python", "giant/workflow/run.py"])
-    b2luigi.set_setting("task_cmd_additional_args", ["--spec", str(spec_path)])
-
-
-def main(argv: list[str] | None = None) -> None:
-    args, _ = build_parser().parse_known_args(argv)
-    spec_path = Path(args.spec).resolve()
-    spec = load_spec(spec_path)
-    configure(spec, spec_path, batch=args.batch)
-
-    kwargs: dict = {}
-    if args.scheduler_host:
-        kwargs["scheduler_host"] = args.scheduler_host
-    if args.scheduler_port:
-        kwargs["scheduler_port"] = args.scheduler_port
-
-    b2luigi.process(
-        WorkflowTask(workflow_name=spec.name),
-        workers=args.workers,
-        batch=args.batch,
-        dry_run=args.mode == "dry-run",
-        show_output=args.mode == "show-output",
-        remove=args.mode == "remove",
-        auto_confirm=args.mode == "remove",
-        # run.py owns --spec/--mode/...; b2luigi must not choke on them.
-        ignore_additional_command_line_args=True,
-        **kwargs,
-    )
-
-
-if __name__ == "__main__":
-    main()

giant/workflow/spec.py

@@ -1,383 +0,0 @@
-"""Workflow TOML -> frozen dataclasses, validation, and per-task spec hashes.
-
-One spec file is the only place a pipeline is parameterised (see
-``configs/workflow_example.toml``):
-
-    [workflow]  name / result_dir / log_dir
-    [condor]    accounting group, repo dir, env script, docker images
-    [dataset]   steps (training) + reference (rollout seeds & analysis truth)
-    [geometry]  geometry-oracle build options
-    [[train]]   one per training run      (name, config, epochs, overrides, ...)
-    [[rollout]] one per rollout           (name, train = <a [[train]].name>, ...)
-    [[analysis]] one per comparison       (name, rollouts = [<[[rollout]].name>, ...])
-
-Every task carries its ``name`` plus a short ``spec_hash`` — 8 hex of the
-canonical JSON of its own resolved sub-spec **including its transitive
-parents**. That is what makes an edited spec produce a fresh result directory
-instead of silently reusing outputs computed under different settings: change
-the dataset and every hash downstream of it changes too.
-
-Unknown keys are rejected (with the valid ones listed), in the same spirit as
-``giant.config.validate_config_keys`` — a typo in a workflow spec would
-otherwise be a silently ignored setting on a multi-day pipeline.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import tomllib
-from dataclasses import MISSING, dataclass, field, fields, is_dataclass
-from pathlib import Path
-from typing import Any
-
-__all__ = [
-    "AnalysisSpec",
-    "CondorSpec",
-    "DatasetSpec",
-    "GeometrySpec",
-    "RolloutSpec",
-    "TrainSpec",
-    "WorkflowSpec",
-    "load_spec",
-    "spec_hash",
-]
-
-
-class WorkflowSpecError(ValueError):
-    """Raised for any malformed workflow spec (unknown key, bad reference, ...)."""
-
-
-# ---------------------------------------------------------------------------
-# sub-specs
-# ---------------------------------------------------------------------------
-
-
-@dataclass(frozen=True)
-class CondorSpec:
-    """Where and how jobs run — the batch-system half of the spec.
-
-    ``repo_dir`` doubles as b2luigi's ``working_dir`` (jobs ``cd`` there before
-    running ``giant/workflow/run.py``), and ``env_script`` is sourced first,
-    since submit and worker machines don't share an environment.
-    """
-
-    accounting_group: str
-    repo_dir: str
-    env_script: str = ""
-    docker_image_cpu: str = "cverstege/alma9-gridjob"
-    docker_image_gpu: str = "mschnepf/slc7-condocker"
-    remote: bool = True
-    request_cpus: int = 1
-    request_memory_mb: int = 8192
-
-
-@dataclass(frozen=True)
-class DatasetSpec:
-    """The two datasets every pipeline needs.
-
-    ``steps`` is what training reads; ``reference`` is the held-out file
-    rollouts are seeded from and the analysis compares against (the "one
-    ground truth" premise of ``giant.analysis``).
-    """
-
-    steps: str
-    reference: str
-
-
-@dataclass(frozen=True)
-class GeometrySpec:
-    """``dwarf build-geometry-oracle`` options (see giant/tools/geometry_oracle.py)."""
-
-    method: str = "slab"
-    k: int = 1
-    subsample: int = 500_000
-    escape_factor: float = 5.0
-    seed: int = 0
-    depth_axis: int = 2
-    n_bins: int = 2000
-
-
-@dataclass(frozen=True)
-class TrainSpec:
-    """One training run, fanned out into ``ceil(epochs / epochs_per_job)`` jobs.
-
-    ``overrides`` are ``[train]``/model config keys merged on top of ``config``
-    exactly as ``giant train``'s flags are (``giant.config.merge_cli_overrides``),
-    so anything expressible on the CLI is expressible here.
-    """
-
-    name: str
-    config: str | None = None
-    epochs: int = 1
-    epochs_per_job: int = 1
-    overrides: dict[str, Any] = field(default_factory=dict)
-    request_gpus: int = 1
-    gpu_type: str | None = None
-    gpu_memory_mb: int | None = None
-    request_memory_mb: int = 16384
-    request_cpus: int = 4
-    walltime_s: int = 86400
-    num_workers: int = 4
-    shuffle_buffer: int = 65536
-    device: str | None = None
-
-
-@dataclass(frozen=True)
-class RolloutSpec:
-    """One ``giant rollout`` run against the checkpoint of ``train``."""
-
-    name: str
-    train: str
-    n_events: int | None = None
-    energy_cutoff: float = 0.1
-    max_steps: int = 1000
-    steps: int = 10
-    batch_size: int = 4096
-    max_tracks_per_event: int | None = None
-    escape_threshold: float | None = None
-    weights: str = "raw"
-    seed: int | None = None
-    request_gpus: int = 1
-    gpu_type: str | None = None
-    gpu_memory_mb: int | None = None
-    request_memory_mb: int = 16384
-    request_cpus: int = 2
-    walltime_s: int = 86400
-    device: str | None = None
-
-
-@dataclass(frozen=True)
-class AnalysisSpec:
-    """One rollout-vs-reference comparison (N rollout series, one reference)."""
-
-    name: str
-    rollouts: tuple[str, ...]
-    chunks: int = 1
-    energy_bins: int = 4
-    bins: int = 50
-    top_pdg: int = 6
-    gallery: bool = False
-    request_memory_mb: int = 8192
-    request_cpus: int = 1
-
-
-@dataclass(frozen=True)
-class WorkflowSpec:
-    """A whole pipeline: the parsed spec file plus name-keyed lookups."""
-
-    name: str
-    result_dir: str
-    log_dir: str
-    condor: CondorSpec
-    dataset: DatasetSpec
-    geometry: GeometrySpec
-    trains: tuple[TrainSpec, ...]
-    rollouts: tuple[RolloutSpec, ...]
-    analyses: tuple[AnalysisSpec, ...]
-    path: str = ""
-
-    # -- lookups ----------------------------------------------------------
-    def train(self, name: str) -> TrainSpec:
-        return _lookup(self.trains, name, "train")
-
-    def rollout(self, name: str) -> RolloutSpec:
-        return _lookup(self.rollouts, name, "rollout")
-
-    def analysis(self, name: str) -> AnalysisSpec:
-        return _lookup(self.analyses, name, "analysis")
-
-    # -- hashes -----------------------------------------------------------
-    # Each one folds in everything upstream of it, so a change anywhere in a
-    # task's ancestry moves its result directory (and only the affected
-    # subtree's).
-    def dataset_hash(self) -> str:
-        return spec_hash(self.dataset)
-
-    def warm_cache_hash(self, train_name: str) -> str:
-        # The setup cache depends on the dataset and on what this training's
-        # config asks of it (val split, conditioning, router) — not on how
-        # many epochs it runs for, so epochs/resources are deliberately left
-        # out and two trainings sharing a config share one warm-cache job.
-        t = self.train(train_name)
-        return spec_hash(self.dataset, t.config, t.overrides)
-
-    def geometry_hash(self) -> str:
-        return spec_hash(self.dataset, self.geometry)
-
-    def train_hash(self, name: str) -> str:
-        return spec_hash(self.dataset, self.train(name))
-
-    def rollout_hash(self, name: str) -> str:
-        ro = self.rollout(name)
-        return spec_hash(self.dataset, self.geometry, self.train(ro.train), ro)
-
-    def analysis_hash(self, name: str) -> str:
-        an = self.analysis(name)
-        parents = [self.rollout(r) for r in an.rollouts]
-        train_parents = [self.train(r.train) for r in parents]
-        return spec_hash(self.dataset, self.geometry, train_parents, parents, an)
-
-
-def _lookup(items, name: str, kind: str):
-    for item in items:
-        if item.name == name:
-            return item
-    known = ", ".join(sorted(i.name for i in items)) or "(none defined)"
-    raise WorkflowSpecError(f"no [[{kind}]] named {name!r} in this workflow — defined: {known}")
-
-
-# ---------------------------------------------------------------------------
-# hashing
-# ---------------------------------------------------------------------------
-
-
-def spec_hash(*parts: Any, length: int = 8) -> str:
-    """Short stable hash of one or more (sub-)specs.
-
-    Canonical JSON (sorted keys, dataclasses expanded) so the value depends
-    only on the resolved settings — not on key order in the TOML, nor on
-    which defaults were written out explicitly.
-    """
-    payload = json.dumps([_canonical(p) for p in parts], sort_keys=True, separators=(",", ":"))
-    return hashlib.sha256(payload.encode()).hexdigest()[:length]
-
-
-def _canonical(value: Any) -> Any:
-    if is_dataclass(value) and not isinstance(value, type):
-        return {f.name: _canonical(getattr(value, f.name)) for f in fields(value)}
-    if isinstance(value, dict):
-        return {str(k): _canonical(v) for k, v in value.items()}
-    if isinstance(value, (list, tuple)):
-        return [_canonical(v) for v in value]
-    if isinstance(value, Path):
-        return str(value)
-    return value
-
-
-# ---------------------------------------------------------------------------
-# parsing
-# ---------------------------------------------------------------------------
-
-
-def _build(cls, data: dict, where: str):
-    """Instantiate a frozen sub-spec, rejecting unknown/missing keys loudly."""
-    valid = {f.name for f in fields(cls)}
-    unknown = sorted(set(data) - valid)
-    if unknown:
-        raise WorkflowSpecError(f"{where}: unknown key(s) {unknown} — valid keys: {sorted(valid)}")
-    required = {f.name for f in fields(cls) if f.default is MISSING and f.default_factory is MISSING}
-    missing = sorted(required - set(data))
-    if missing:
-        raise WorkflowSpecError(f"{where}: missing required key(s) {missing}")
-    return cls(**data)
-
-
-def load_spec(path: str | Path) -> WorkflowSpec:
-    """Parse and validate a workflow TOML file."""
-    path = Path(path)
-    try:
-        raw = tomllib.loads(path.read_text())
-    except tomllib.TOMLDecodeError as exc:
-        raise WorkflowSpecError(f"{path}: not valid TOML — {exc}") from exc
-    return parse_spec(raw, path=path)
-
-
-def parse_spec(raw: dict, path: str | Path = "") -> WorkflowSpec:
-    """Validate an already-parsed workflow spec mapping."""
-    top_valid = {"workflow", "condor", "dataset", "geometry", "train", "rollout", "analysis"}
-    unknown = sorted(set(raw) - top_valid)
-    if unknown:
-        raise WorkflowSpecError(
-            f"{path or '<spec>'}: unknown top-level table(s) {unknown} — valid: {sorted(top_valid)}"
-        )
-
-    for required in ("workflow", "condor", "dataset"):
-        if required not in raw:
-            raise WorkflowSpecError(f"{path or '<spec>'}: missing required [{required}] table")
-
-    wf = dict(raw["workflow"])
-    wf_valid = {"name", "result_dir", "log_dir"}
-    wf_unknown = sorted(set(wf) - wf_valid)
-    if wf_unknown:
-        raise WorkflowSpecError(f"[workflow]: unknown key(s) {wf_unknown} — valid keys: {sorted(wf_valid)}")
-    if "name" not in wf or "result_dir" not in wf:
-        raise WorkflowSpecError("[workflow]: 'name' and 'result_dir' are required")
-    result_dir = str(Path(wf["result_dir"]).expanduser())
-    log_dir = str(Path(wf.get("log_dir", Path(result_dir) / "logs")).expanduser())
-
-    condor = _build(CondorSpec, dict(raw["condor"]), "[condor]")
-    dataset = _build(DatasetSpec, dict(raw["dataset"]), "[dataset]")
-    geometry = _build(GeometrySpec, dict(raw.get("geometry", {})), "[geometry]")
-
-    trains = tuple(_build(TrainSpec, dict(t), f"[[train]] #{i}") for i, t in enumerate(raw.get("train", [])))
-    rollouts = tuple(_build(RolloutSpec, dict(r), f"[[rollout]] #{i}") for i, r in enumerate(raw.get("rollout", [])))
-    analyses = tuple(
-        _build(AnalysisSpec, {**a, "rollouts": tuple(a.get("rollouts", ()))}, f"[[analysis]] #{i}")
-        for i, a in enumerate(raw.get("analysis", []))
-    )
-
-    _check_unique(trains, "train")
-    _check_unique(rollouts, "rollout")
-    _check_unique(analyses, "analysis")
-
-    train_names = {t.name for t in trains}
-    for ro in rollouts:
-        if ro.train not in train_names:
-            raise WorkflowSpecError(
-                f"[[rollout]] {ro.name!r}: train={ro.train!r} names no [[train]] — defined: {sorted(train_names)}"
-            )
-    rollout_names = {r.name for r in rollouts}
-    for an in analyses:
-        if not an.rollouts:
-            raise WorkflowSpecError(f"[[analysis]] {an.name!r}: 'rollouts' must name at least one [[rollout]]")
-        for r in an.rollouts:
-            if r not in rollout_names:
-                raise WorkflowSpecError(
-                    f"[[analysis]] {an.name!r}: rollout {r!r} is not defined — "
-                    f"defined: {sorted(rollout_names) or '(none)'}"
-                )
-        if len(set(an.rollouts)) != len(an.rollouts):
-            raise WorkflowSpecError(f"[[analysis]] {an.name!r}: repeated rollout name(s) in 'rollouts'")
-        if an.chunks < 1:
-            raise WorkflowSpecError(f"[[analysis]] {an.name!r}: chunks must be >= 1, got {an.chunks}")
-
-    for t in trains:
-        if t.epochs < 1:
-            raise WorkflowSpecError(f"[[train]] {t.name!r}: epochs must be >= 1, got {t.epochs}")
-        if t.epochs_per_job < 1:
-            raise WorkflowSpecError(f"[[train]] {t.name!r}: epochs_per_job must be >= 1, got {t.epochs_per_job}")
-
-    return WorkflowSpec(
-        name=wf["name"],
-        result_dir=result_dir,
-        log_dir=log_dir,
-        condor=condor,
-        dataset=dataset,
-        geometry=geometry,
-        trains=trains,
-        rollouts=rollouts,
-        analyses=analyses,
-        path=str(path),
-    )
-
-
-def _check_unique(items, kind: str) -> None:
-    names = [i.name for i in items]
-    dupes = sorted({n for n in names if names.count(n) > 1})
-    if dupes:
-        raise WorkflowSpecError(f"[[{kind}]] names must be unique — repeated: {dupes}")
-
-
-def epoch_milestones(train: TrainSpec) -> list[int]:
-    """Cumulative epoch counts, one per chained ``TrainEpochTask``.
-
-    ``epochs_per_job`` trades queue waits against job length: with
-    ``epochs=10, epochs_per_job=3`` this is ``[3, 6, 9, 10]``, i.e. job *k*
-    resumes job *k-1*'s ``last.pt`` and trains up to its own milestone.
-    """
-    step = train.epochs_per_job
-    milestones = list(range(step, train.epochs + 1, step))
-    if not milestones or milestones[-1] != train.epochs:
-        milestones.append(train.epochs)
-    return milestones

giant/workflow/tasks.py

@@ -1,647 +0,0 @@
-"""The b2luigi task graph: cache-warm -> train -> rollout -> analysis.
-
-    DatasetTask (external)  ─┬─> WarmCacheTask(train) ─> TrainEpochTask(train, 1..N) ─> TrainTask(train) ─┐
-                             └─> GeometryOracleTask ──┐                                                   │
-                                                      └──> RolloutTask(rollout) <────────────────────────┘
-                                                                       │
-    AnalysisPrepTask(analysis) ─> AnalysisComputeTask(analysis, plot, chunk) ─> AnalysisRenderTask(analysis)
-                                                                                        ^
-    WorkflowTask (wrapper) ─────────────────────────────────────────────────────────────┘
-
-Every task's output directory is ``<result_dir>/<kind>/name=<name>/spec_hash=
-<hash>/…`` — the hash covers the task's resolved sub-spec *and its transitive
-parents* (``giant/workflow/spec.py``), so editing the spec produces a fresh
-directory for exactly the affected subtree instead of silently reusing stale
-outputs.
-
-Task bodies never reimplement anything: they call the same entry points the
-CLIs do (``run_warm_setup_cache``, ``run_build_geometry_oracle``,
-``run_train_job``, ``giant.analysis.prep``/``compute_one``/``merge_all``,
-``render_run``), or shell out to ``giant rollout``, which has no library-level
-entry point of its own.
-
-Training is fanned out into **one short GPU job per epoch** (or per
-``epochs_per_job`` epochs): job *k* runs ``run_train_job`` with ``epochs = k``
-and ``resume = <job k-1>/last.pt``, which the training loop already handles
-(``giant/training/loop.py`` sets ``start_epoch = ckpt["epoch"] + 1`` and
-returns early when the checkpoint already covers ``epochs``). A 200-epoch run
-then becomes 200 schedulable jobs that survive preemption and give luigi a
-real progress signal, at the cost of one (cache-warmed) setup scan and one
-queue wait per job.
-"""
-
-from __future__ import annotations
-
-import json
-import shutil
-import subprocess
-import sys
-from pathlib import Path
-
-import b2luigi
-
-from giant.workflow.htcondor import cpu_settings, gpu_settings
-from giant.workflow.spec import WorkflowSpec, epoch_milestones
-
-__all__ = [
-    "AnalysisComputeTask",
-    "AnalysisPrepTask",
-    "AnalysisRenderTask",
-    "DatasetTask",
-    "GeometryOracleTask",
-    "RolloutTask",
-    "TrainEpochTask",
-    "TrainTask",
-    "WarmCacheTask",
-    "WorkflowTask",
-    "analysis_dir",
-    "analysis_jobs",
-    "get_spec",
-    "set_spec",
-]
-
-
-# ---------------------------------------------------------------------------
-# the active spec
-# ---------------------------------------------------------------------------
-# luigi parameters must be simple scalars, so tasks carry only `name` +
-# `spec_hash` and read the rest out of the one spec this process was started
-# with. Batch workers re-execute `run.py --spec <same file>` (see
-# `task_cmd_additional_args` there), so they resolve the identical spec.
-
-_SPEC: WorkflowSpec | None = None
-
-
-def set_spec(spec: WorkflowSpec) -> None:
-    global _SPEC
-    _SPEC = spec
-
-
-def get_spec() -> WorkflowSpec:
-    if _SPEC is None:
-        raise RuntimeError("no workflow spec loaded — call giant.workflow.tasks.set_spec() first")
-    return _SPEC
-
-
-def _result_dir(*parts: str) -> Path:
-    return Path(get_spec().result_dir).joinpath(*parts)
-
-
-def _task_dir(kind: str, name: str, spec_hash: str) -> Path:
-    """``<result_dir>/<kind>/name=<name>/spec_hash=<hash>``."""
-    return _result_dir(kind, f"name={name}", f"spec_hash={spec_hash}")
-
-
-def analysis_dir(spec: WorkflowSpec, name: str) -> Path:
-    """The analysis run directory — what ``prep`` lays out and every later step reads."""
-    return Path(spec.result_dir) / "analysis" / f"name={name}" / f"spec_hash={spec.analysis_hash(name)}"
-
-
-def analysis_jobs(spec: WorkflowSpec, name: str) -> list[tuple[str, int]]:
-    """Every ``(plot_id, chunk)`` compute job of one analysis.
-
-    ``chunkable=False`` specs (the checkpoint-bound diagnostics, already
-    bounded/subsampled) always run as a single chunk — the same rule the
-    deleted ``_job_walltimes`` applied.
-    """
-    from giant.analysis.catalog import catalog_ids, get_spec as get_plot_spec
-
-    chunks = spec.analysis(name).chunks
-    jobs: list[tuple[str, int]] = []
-    for plot_id in catalog_ids():
-        n = chunks if get_plot_spec(plot_id).chunkable else 1
-        jobs.extend((plot_id, chunk) for chunk in range(n))
-    return jobs
-
-
-def _giant_cmd() -> list[str]:
-    """How to invoke the ``giant`` CLI from inside a task (worker or locally)."""
-    return [sys.executable, "-m", "giant.cli"]
-
-
-# ---------------------------------------------------------------------------
-# inputs
-# ---------------------------------------------------------------------------
-
-
-class DatasetTask(b2luigi.ExternalTask):
-    """A steps parquet file or directory that must already exist.
-
-    Nothing produces it, so a missing path is a hard, immediate error rather
-    than a job that fails hours later — the usual cause being ``/ceph`` not
-    mounted on the machine the workflow was started from.
-    """
-
-    path = b2luigi.Parameter()
-
-    def output(self):
-        return b2luigi.LocalTarget(str(self.path))
-
-    def complete(self):
-        if not Path(str(self.path)).exists():
-            raise FileNotFoundError(
-                f"dataset {self.path!r} does not exist — is /ceph mounted on this machine? "
-                "(see CLAUDE.md's Compute environment section)"
-            )
-        return True
-
-
-# ---------------------------------------------------------------------------
-# setup stage
-# ---------------------------------------------------------------------------
-
-
-class WarmCacheTask(b2luigi.Task):
-    """Precompute one training's setup-stage sidecar (vocab maps, event split,
-    normalizer stats) so every per-epoch job is a cache hit instead of a
-    full rescan.
-
-    The real product (``<data>.giant_train_cache.json``) lives next to the
-    dataset, not under ``result_dir``, so the target here is a small stamp
-    recording that sidecar's path/mtime/size.
-    """
-
-    name = b2luigi.Parameter()
-    spec_hash = b2luigi.Parameter()
-
-    @property
-    def htcondor_settings(self):
-        spec = get_spec()
-        return cpu_settings(spec.condor, request_memory_mb=32768, request_cpus=4, walltime_s=21600)
-
-    def requires(self):
-        yield DatasetTask(path=get_spec().dataset.steps)
-
-    def output(self):
-        return b2luigi.LocalTarget(str(_task_dir("warm_cache", str(self.name), str(self.spec_hash)) / "stamp.json"))
-
-    def run(self):
-        from giant.data.setup_cache import sidecar_path
-        from giant.tools.warm_setup_cache import run_warm_setup_cache
-
-        spec = get_spec()
-        train = spec.train(str(self.name))
-        run_warm_setup_cache(
-            data=spec.dataset.steps,
-            config_path=Path(train.config) if train.config else None,
-        )
-        sidecar = Path(sidecar_path(spec.dataset.steps))
-        stamp = {
-            "sidecar": str(sidecar),
-            "mtime": sidecar.stat().st_mtime if sidecar.exists() else None,
-            "size": sidecar.stat().st_size if sidecar.exists() else None,
-        }
-        out = Path(self.output().path)
-        out.parent.mkdir(parents=True, exist_ok=True)
-        out.write_text(json.dumps(stamp, indent=2))
-
-
-class GeometryOracleTask(b2luigi.Task):
-    """Build the position -> (material, layer_id) oracle every rollout needs."""
-
-    spec_hash = b2luigi.Parameter()
-
-    @property
-    def htcondor_settings(self):
-        spec = get_spec()
-        return cpu_settings(spec.condor, request_memory_mb=32768, request_cpus=4, walltime_s=21600)
-
-    def requires(self):
-        yield DatasetTask(path=get_spec().dataset.steps)
-
-    def output(self):
-        return b2luigi.LocalTarget(
-            str(_result_dir("geometry", f"spec_hash={self.spec_hash}") / "oracle.pkl"),
-        )
-
-    def run(self):
-        from giant.tools.geometry_oracle import run_build_geometry_oracle
-
-        spec = get_spec()
-        g = spec.geometry
-        out = Path(self.output().path)
-        out.parent.mkdir(parents=True, exist_ok=True)
-        run_build_geometry_oracle(
-            data=Path(spec.dataset.steps),
-            out=out,
-            method=g.method,
-            k=g.k,
-            subsample=g.subsample,
-            escape_factor=g.escape_factor,
-            seed=g.seed,
-            depth_axis=g.depth_axis,
-            n_bins=g.n_bins,
-        )
-
-
-# ---------------------------------------------------------------------------
-# training
-# ---------------------------------------------------------------------------
-
-
-def _train_cfg(spec: WorkflowSpec, name: str, epochs: int) -> dict:
-    """The merged config one training job runs, resolved exactly as `giant train` does."""
-    from giant import config as gconfig
-
-    train = spec.train(name)
-    flags = {**train.overrides, "epochs": epochs}
-    overrides = gconfig.overrides_from_flags(flags)
-    cfg = gconfig.merge_cli_overrides(
-        gconfig.DEFAULT_CONFIG,
-        Path(train.config) if train.config else None,
-        overrides,
-    )
-    gconfig.validate_config(cfg, resume=True)
-    return cfg
-
-
-class TrainEpochTask(b2luigi.Task):
-    """Epochs up to ``milestone`` of one training, resuming the previous job.
-
-    Target is ``last.pt``. ``best.pt`` is written by the loop *only when that
-    epoch improved*, and ``best_val_loss`` travels inside the checkpoint, so
-    the global best comparison stays correct across jobs: "``best.pt`` exists
-    in milestone dir *k*" means exactly "one of that job's epochs was the best
-    so far".
-    """
-
-    name = b2luigi.Parameter()
-    spec_hash = b2luigi.Parameter()
-    milestone = b2luigi.IntParameter()
-
-    @property
-    def htcondor_settings(self):
-        spec = get_spec()
-        train = spec.train(str(self.name))
-        return gpu_settings(
-            spec.condor,
-            request_gpus=train.request_gpus,
-            gpu_type=train.gpu_type,
-            gpu_memory_mb=train.gpu_memory_mb,
-            request_memory_mb=train.request_memory_mb,
-            request_cpus=train.request_cpus,
-            walltime_s=train.walltime_s,
-        )
-
-    @property
-    def _dir(self) -> Path:
-        return _task_dir("train_epoch", str(self.name), str(self.spec_hash)) / f"epochs={int(self.milestone)}"
-
-    def _previous_milestone(self) -> int | None:
-        spec = get_spec()
-        milestones = epoch_milestones(spec.train(str(self.name)))
-        index = milestones.index(int(self.milestone))
-        return milestones[index - 1] if index > 0 else None
-
-    def requires(self):
-        previous = self._previous_milestone()
-        if previous is None:
-            yield WarmCacheTask(name=self.name, spec_hash=get_spec().warm_cache_hash(str(self.name)))
-        else:
-            yield TrainEpochTask(name=self.name, spec_hash=self.spec_hash, milestone=previous)
-
-    def output(self):
-        return b2luigi.LocalTarget(str(self._dir / "last.pt"))
-
-    def run(self):
-        import torch
-
-        from giant import config as gconfig
-        from giant.pipeline import run_train_job
-
-        spec = get_spec()
-        train = spec.train(str(self.name))
-        cfg = _train_cfg(spec, str(self.name), int(self.milestone))
-
-        previous = self._previous_milestone()
-        resume = None
-        if previous is not None:
-            resume = _task_dir("train_epoch", str(self.name), str(self.spec_hash)) / f"epochs={previous}" / "last.pt"
-
-        device = torch.device(train.device) if train.device else gconfig.auto_device()
-        out_dir = self._dir
-        out_dir.mkdir(parents=True, exist_ok=True)
-        run_train_job(
-            data=Path(spec.dataset.steps),
-            cfg=cfg,
-            out_dir=out_dir,
-            device=device,
-            shuffle_buffer=train.shuffle_buffer,
-            num_workers=train.num_workers,
-            resume=resume,
-            cache_setup=True,
-        )
-
-
-class TrainTask(b2luigi.Task):
-    """Publish one training's canonical outputs, hiding the epoch fan-out.
-
-    Everything downstream (``RolloutTask``, humans, ``giant analyze metrics``)
-    points here and never has to know which milestone directory happened to
-    hold the best checkpoint.
-    """
-
-    name = b2luigi.Parameter()
-    spec_hash = b2luigi.Parameter()
-    batch_system = "local"
-
-    @property
-    def _milestones(self) -> list[int]:
-        return epoch_milestones(get_spec().train(str(self.name)))
-
-    def requires(self):
-        yield TrainEpochTask(name=self.name, spec_hash=self.spec_hash, milestone=self._milestones[-1])
-
-    @property
-    def _dir(self) -> Path:
-        return _task_dir("train", str(self.name), str(self.spec_hash))
-
-    def output(self):
-        d = self._dir
-        return {
-            "best.pt": b2luigi.LocalTarget(str(d / "best.pt")),
-            "last.pt": b2luigi.LocalTarget(str(d / "last.pt")),
-            "metrics.csv": b2luigi.LocalTarget(str(d / "metrics.csv")),
-        }
-
-    def run(self):
-        epoch_base = _task_dir("train_epoch", str(self.name), str(self.spec_hash))
-        milestone_dirs = [epoch_base / f"epochs={m}" for m in self._milestones]
-
-        best_dirs = [d for d in milestone_dirs if (d / "best.pt").exists()]
-        if not best_dirs:
-            raise FileNotFoundError(
-                f"no best.pt in any milestone directory under {epoch_base} — "
-                "did every epoch job run with a validation split?"
-            )
-        out = self._dir
-        out.mkdir(parents=True, exist_ok=True)
-        shutil.copy2(best_dirs[-1] / "best.pt", out / "best.pt")
-        shutil.copy2(milestone_dirs[-1] / "last.pt", out / "last.pt")
-        for extra in ("config.toml", "run_meta.json"):
-            src = milestone_dirs[-1] / extra
-            if src.exists():
-                shutil.copy2(src, out / extra)
-
-        # One metrics.csv for the whole run: the first job's header, then
-        # every job's rows in epoch order, so `giant analyze metrics` sees a
-        # single continuous training curve.
-        lines: list[str] = []
-        header: str | None = None
-        for d in milestone_dirs:
-            csv = d / "metrics.csv"
-            if not csv.exists():
-                continue
-            rows = csv.read_text().splitlines()
-            if not rows:
-                continue
-            if header is None:
-                header = rows[0]
-            lines.extend(rows[1:])
-        (out / "metrics.csv").write_text("\n".join([header or ""] + lines) + "\n")
-
-
-# ---------------------------------------------------------------------------
-# rollout
-# ---------------------------------------------------------------------------
-
-
-class RolloutTask(b2luigi.Task):
-    """Roll one trained checkpoint forward into full showers.
-
-    ``giant rollout`` has no library-level entry point, so this shells out to
-    the CLI — with an explicit ``--out``, which puts the YAML sidecar at the
-    deterministic ``rollout.yaml`` next to the parquet (see
-    ``giant/cli.py:_write_prediction_ref``).
-    """
-
-    name = b2luigi.Parameter()
-    spec_hash = b2luigi.Parameter()
-
-    @property
-    def htcondor_settings(self):
-        spec = get_spec()
-        ro = spec.rollout(str(self.name))
-        return gpu_settings(
-            spec.condor,
-            request_gpus=ro.request_gpus,
-            gpu_type=ro.gpu_type,
-            gpu_memory_mb=ro.gpu_memory_mb,
-            request_memory_mb=ro.request_memory_mb,
-            request_cpus=ro.request_cpus,
-            walltime_s=ro.walltime_s,
-        )
-
-    @property
-    def _dir(self) -> Path:
-        return _task_dir("rollout", str(self.name), str(self.spec_hash))
-
-    def requires(self):
-        spec = get_spec()
-        ro = spec.rollout(str(self.name))
-        yield TrainTask(name=ro.train, spec_hash=spec.train_hash(ro.train))
-        yield GeometryOracleTask(spec_hash=spec.geometry_hash())
-        yield DatasetTask(path=spec.dataset.reference)
-
-    def output(self):
-        d = self._dir
-        return {
-            "rollout.parquet": b2luigi.LocalTarget(str(d / "rollout.parquet")),
-            "rollout.yaml": b2luigi.LocalTarget(str(d / "rollout.yaml")),
-        }
-
-    def run(self):
-        spec = get_spec()
-        ro = spec.rollout(str(self.name))
-        out = self._dir / "rollout.parquet"
-        out.parent.mkdir(parents=True, exist_ok=True)
-
-        checkpoint = _task_dir("train", ro.train, spec.train_hash(ro.train)) / "best.pt"
-        oracle = _result_dir("geometry", f"spec_hash={spec.geometry_hash()}") / "oracle.pkl"
-
-        cmd = [
-            *_giant_cmd(),
-            "rollout",
-            spec.dataset.reference,
-            "--checkpoint",
-            str(checkpoint),
-            "--geometry",
-            str(oracle),
-            "--out",
-            str(out),
-            "--energy-cutoff",
-            str(ro.energy_cutoff),
-            "--max-steps",
-            str(ro.max_steps),
-            "--steps",
-            str(ro.steps),
-            "--batch-size",
-            str(ro.batch_size),
-            "--weights",
-            ro.weights,
-        ]
-        for flag, value in (
-            ("--n-events", ro.n_events),
-            ("--max-tracks-per-event", ro.max_tracks_per_event),
-            ("--escape-threshold", ro.escape_threshold),
-            ("--seed", ro.seed),
-            ("--device", ro.device),
-        ):
-            if value is not None:
-                cmd += [flag, str(value)]
-        subprocess.run(cmd, check=True)
-
-
-# ---------------------------------------------------------------------------
-# analysis
-# ---------------------------------------------------------------------------
-
-
-class AnalysisPrepTask(b2luigi.Task):
-    """Resolve the shared bin edges/group sets once, for every compute job.
-
-    Cheap and streaming, so it runs locally: everything after it needs
-    ``shared.json``/``run_meta.json`` to already exist.
-    """
-
-    name = b2luigi.Parameter()
-    spec_hash = b2luigi.Parameter()
-    batch_system = "local"
-
-    def requires(self):
-        spec = get_spec()
-        for rollout_name in spec.analysis(str(self.name)).rollouts:
-            yield RolloutTask(name=rollout_name, spec_hash=spec.rollout_hash(rollout_name))
-
-    @property
-    def _dir(self) -> Path:
-        return analysis_dir(get_spec(), str(self.name))
-
-    def output(self):
-        d = self._dir
-        return {
-            "shared.json": b2luigi.LocalTarget(str(d / "shared.json")),
-            "run_meta.json": b2luigi.LocalTarget(str(d / "run_meta.json")),
-        }
-
-    def run(self):
-        from giant.analysis import prep
-
-        spec = get_spec()
-        an = spec.analysis(str(self.name))
-        yamls = [_task_dir("rollout", r, spec.rollout_hash(r)) / "rollout.yaml" for r in an.rollouts]
-        prep(
-            yamls,
-            run_dir=self._dir,
-            n_chunks=an.chunks,
-            labels=list(an.rollouts),
-            n_energy_bins=an.energy_bins,
-            n_marginal_bins=an.bins,
-            top_k_pdg=an.top_pdg,
-        )
-
-
-class AnalysisComputeTask(b2luigi.Task):
-    """One (plot, chunk) streaming reduction — the replaced ``jobs.txt`` row.
-
-    The output path is the on-disk contract ``compute-one``/``merge_one``
-    already share (``reduced_partial/<id>__<chunk>.json``), declared
-    explicitly rather than through b2luigi's own output naming so that
-    contract is untouched.
-    """
-
-    name = b2luigi.Parameter()
-    spec_hash = b2luigi.Parameter()
-    plot_id = b2luigi.Parameter()
-    chunk = b2luigi.IntParameter()
-
-    @property
-    def htcondor_settings(self):
-        # A property, so it is evaluated at submit time — i.e. after prep has
-        # written run_meta.json, whose row counts size the walltime request.
-        from giant.analysis import RunMeta
-        from giant.analysis.runtime_estimate import estimate_runtime_s
-
-        spec = get_spec()
-        an = spec.analysis(str(self.name))
-        walltime = None
-        meta_path = analysis_dir(spec, str(self.name)) / "run_meta.json"
-        if meta_path.exists():
-            from giant.analysis.catalog import get_spec as get_plot_spec
-
-            meta = RunMeta.load(meta_path)
-            chunkable = get_plot_spec(str(self.plot_id)).chunkable
-            n_rows = meta.rows_per_chunk[int(self.chunk)] if chunkable and meta.rows_per_chunk else meta.total_rows
-            walltime = estimate_runtime_s(str(self.plot_id), n_rows)
-        return cpu_settings(
-            spec.condor,
-            request_memory_mb=an.request_memory_mb,
-            request_cpus=an.request_cpus,
-            walltime_s=walltime,
-        )
-
-    def requires(self):
-        yield AnalysisPrepTask(name=self.name, spec_hash=self.spec_hash)
-
-    def output(self):
-        run_dir = analysis_dir(get_spec(), str(self.name))
-        return b2luigi.LocalTarget(str(run_dir / "reduced_partial" / f"{self.plot_id}__{int(self.chunk)}.json"))
-
-    def run(self):
-        from giant.analysis import compute_one
-
-        compute_one(str(self.plot_id), analysis_dir(get_spec(), str(self.name)), chunk_index=int(self.chunk))
-
-
-class AnalysisRenderTask(b2luigi.Task):
-    """Merge every plot's chunk partials, then render the PDFs + gallery.
-
-    Always local — this is the only step that imports plotstyle/LaTeX, which
-    the compute worker images don't have.
-    """
-
-    name = b2luigi.Parameter()
-    spec_hash = b2luigi.Parameter()
-    batch_system = "local"
-
-    def requires(self):
-        spec = get_spec()
-        for plot_id, chunk in analysis_jobs(spec, str(self.name)):
-            yield AnalysisComputeTask(
-                name=self.name,
-                spec_hash=self.spec_hash,
-                plot_id=plot_id,
-                chunk=chunk,
-            )
-
-    def output(self):
-        run_dir = analysis_dir(get_spec(), str(self.name))
-        return b2luigi.LocalTarget(str(run_dir / "plots" / "metadata.yaml"))
-
-    def run(self):
-        # render_run joins every plot's chunk partials (merge_all) before
-        # rendering, so this one call is the whole merge+render step.
-        from giant.analysis.render import render_run
-
-        spec = get_spec()
-        render_run(analysis_dir(spec, str(self.name)), run_gallery=spec.analysis(str(self.name)).gallery)
-
-
-class WorkflowTask(b2luigi.WrapperTask):
-    """The whole pipeline: every analysis in the spec, rendered."""
-
-    workflow_name = b2luigi.Parameter()
-
-    def requires(self):
-        spec = get_spec()
-        if not spec.analyses:
-            # A spec with no [[analysis]] still has work to do — fall back to
-            # the deepest tasks it does define.
-            for ro in spec.rollouts:
-                yield RolloutTask(name=ro.name, spec_hash=spec.rollout_hash(ro.name))
-            if not spec.rollouts:
-                for tr in spec.trains:
-                    yield TrainTask(name=tr.name, spec_hash=spec.train_hash(tr.name))
-            return
-        for an in spec.analyses:
-            yield AnalysisRenderTask(name=an.name, spec_hash=spec.analysis_hash(an.name))

pyproject.toml

@@ -1,12 +1,12 @@
 [project]
 name = "giant"
-version = "0.4.0"
+version = "0.3.16"
 description = "Geant4 step-function surrogate via conditional flow matching"
 readme = "README.md"
 requires-python = ">=3.12"
 dependencies = [
     "numpy>=1.26,<3",
-    "pandas>=2.2,<4",
+    "polars>=1.0,<2",
     "pyarrow>=16,<25",
     "tqdm>=4.60,<5",
     "typer>=0.12,<1",
@@ -28,7 +28,11 @@ dev = [
     "ty>=0.0.50,<0.1",
     "bump-my-version>=1.2,<2",
     "git-cliff>=2,<3",
-    "giant[convert,analysis,geometry,wandb,workflow]",
+    # Only used by test fixtures (writing small parquet files) — not a
+    # runtime dependency of giant itself since the pandas -> polars
+    # data-loading rewrite.
+    "pandas>=2.2,<4",
+    "giant[convert,analysis,geometry,wandb]",
 ]
 geometry = [
     "scikit-learn>=1.4,<2",
@@ -49,11 +53,6 @@ analysis = [
     # `giant analyze render` step imports it; compute workers never do.
     "plotstyle>=1.0.0",
 ]
-# b2luigi pulls luigi + tenacity; the only sanctioned way to chain a
-# multi-step pipeline (see giant/workflow/).
-workflow = [
-    "b2luigi>=1.0,<2",
-]
 
 [project.scripts]
 giant = "giant.cli:app"
@@ -108,12 +107,3 @@ explicit = true
 name = "larsbogner"
 url = "https://git.larsbogner.de/api/packages/lars/pypi/simple/"
 explicit = true
-
-# luigi builds task constructors from class-level Parameter descriptors, so a
-# static checker sees no keyword parameters at all on `Task(name=..., ...)`.
-# The workflow code is written against that API; nothing else in the repo is.
-[[tool.ty.overrides]]
-include = ["giant/workflow/**", "tests/test_workflow_tasks.py"]
-
-[tool.ty.overrides.rules]
-unknown-argument = "ignore"

tests/test_condor.py

@@ -1,7 +1,8 @@
-"""Tests for the rollout-YAML(s) → run-directory flow, compute, and merge."""
+"""Tests for the rollout-YAML(s) → run-directory flow, compute, and submit."""
 
 from __future__ import annotations
 
+import sys
 from pathlib import Path
 
 import pyarrow.parquet as pq
@@ -10,6 +11,8 @@ import yaml
 
 from giant.analysis import (
     RunMeta,
+    SubmitConfig,
+    catalog_ids,
     compute_one,
     compute_reduced,
     derive_run_dir,
@@ -17,8 +20,10 @@ from giant.analysis import (
     load_rollout_yamls,
     merge_one,
     prep,
+    write_submit,
 )
-from giant.analysis.run import Context
+from giant.analysis.catalog import get_spec
+from giant.analysis.condor import Context
 from giant.analysis.reduced import Partial, Reduced
 from giant.constants import PREDICT_COORD_METADATA_KEY, ROLLOUT_COORD_VALUE
 from tests.test_analysis_reduce import _reference_frame, _rollout_frame
@@ -81,6 +86,14 @@ def _write_two_inputs(tmp_path: Path) -> tuple[Path, Path]:
     return paths[0], paths[1]
 
 
+def _fake_venv(repo_dir: Path) -> None:
+    """Stand in for a `uv sync`'d venv: write_submit checks `.venv/bin/giant` exists."""
+    giant = repo_dir / ".venv" / "bin" / "giant"
+    giant.parent.mkdir(parents=True, exist_ok=True)
+    giant.write_text("#!/bin/bash\n")
+    giant.chmod(0o755)
+
+
 def _prep(rollout_yamls, run_dir: str | Path | None = None, chunks: int = 1, labels=None) -> Path:
     """``prep`` with small test-sized context bins/sampling."""
     return prep(
@@ -316,6 +329,73 @@ def test_compute_reduced_rejects_out_of_range_chunk(tmp_path: Path):
         compute_one("marginal_edep", run_dir, chunk_index=1)
 
 
+def test_write_submit_description(tmp_path: Path):
+    run_dir = _prep([_write_inputs(tmp_path)])
+    _fake_venv(tmp_path)
+    cfg = SubmitConfig(run_dir=run_dir, accounting_group="cms", repo_dir=tmp_path)
+    txt = write_submit(cfg).read_text()
+    assert "universe = docker" in txt
+    assert "docker_image = cverstege/alma9-gridjob" in txt
+    assert "requirements = TARGET.ProvidesETPResources" in txt
+    assert "accounting_group = cms" in txt
+    assert "+RequestWalltime = $(walltime)" in txt
+    assert "queue plotid,chunk,walltime from" in txt
+    jobs = [line.split(",") for line in (run_dir / "jobs.txt").read_text().split()]
+    assert [i for i, _, _ in jobs] == catalog_ids()
+    assert all(k == "0" for _, k, _ in jobs)  # n_chunks=1 default
+    assert all(int(w) > 0 for _, _, w in jobs)
+    wrapper = run_dir / "run_compute.sh"
+    assert wrapper.exists() and (wrapper.stat().st_mode & 0o111)
+    body = wrapper.read_text()
+    assert "giant analyze compute-one --id" in body
+    assert "--chunk" in body and "--run-dir" in body
+
+
+def test_write_submit_requires_synced_venv(tmp_path: Path, monkeypatch: pytest.MonkeyPatch):
+    run_dir = _prep([_write_inputs(tmp_path)])
+    cfg = SubmitConfig(run_dir=run_dir, accounting_group="cms", repo_dir=tmp_path)
+    # No `giant` next to the (fake) active interpreter, so this falls through
+    # to repo_dir/.venv/bin/giant, which _write_inputs/_prep also didn't create.
+    monkeypatch.setattr(sys, "executable", str(tmp_path / "not-a-venv" / "bin" / "python"))
+    with pytest.raises(FileNotFoundError, match="uv sync"):
+        write_submit(cfg)
+
+
+def test_write_submit_remote_flag(tmp_path: Path):
+    run_dir = _prep([_write_inputs(tmp_path)])
+    _fake_venv(tmp_path)
+    cfg = SubmitConfig(run_dir=run_dir, accounting_group="cms", repo_dir=tmp_path, remote=True)
+    txt = write_submit(cfg).read_text()
+    assert "+RemoteJob = True" in txt
+    assert "ProvidesETPResources" not in txt
+
+
+def test_write_submit_chunks_respect_chunkable(tmp_path: Path):
+    assert get_spec("router_gating").chunkable is False
+    run_dir = _prep([_write_inputs(tmp_path)], chunks=4)
+    _fake_venv(tmp_path)
+    cfg = SubmitConfig(run_dir=run_dir, accounting_group="cms", repo_dir=tmp_path, n_chunks=4)
+    write_submit(cfg)
+    jobs = [line.split(",") for line in (run_dir / "jobs.txt").read_text().split()]
+    counts: dict[str, int] = {}
+    for spec_id, _, _ in jobs:
+        counts[spec_id] = counts.get(spec_id, 0) + 1
+    assert counts["marginal_edep"] == 4
+    assert counts["router_gating"] == 1  # chunkable=False, ignores n_chunks
+
+
+def test_write_submit_rejects_n_chunks_mismatch_with_run_meta(tmp_path: Path):
+    """cfg.n_chunks must match the n_chunks the run_dir was actually prepped
+    with — RunMeta.rows_per_chunk is sized to the prepped value, so a
+    mismatch would otherwise surface as a confusing IndexError deep inside
+    _job_walltimes instead of a clear error here."""
+    run_dir = _prep([_write_inputs(tmp_path)], chunks=2)
+    _fake_venv(tmp_path)
+    cfg = SubmitConfig(run_dir=run_dir, accounting_group="cms", repo_dir=tmp_path, n_chunks=4)
+    with pytest.raises(ValueError, match="n_chunks"):
+        write_submit(cfg)
+
+
 def test_estimate_runtime_s_scales_with_rows_and_margin():
     from giant.analysis import RUNTIME_SAFETY_MARGIN, estimate_runtime_s
     from giant.analysis.runtime_estimate import _FIXED_OVERHEAD_S
@@ -325,3 +405,18 @@ def test_estimate_runtime_s_scales_with_rows_and_margin():
     large = estimate_runtime_s("marginal_edep", 100_000_000)
     assert small >= (1 + RUNTIME_SAFETY_MARGIN) * _FIXED_OVERHEAD_S
     assert large > small  # bigger chunk -> longer estimate
+
+
+def test_write_submit_walltime_grows_with_chunk_rows(tmp_path: Path):
+    """A chunked run's later job walltimes track that chunk's row count."""
+    from giant.analysis.runtime_estimate import estimate_runtime_s
+
+    run_dir = _prep([_write_inputs(tmp_path)], chunks=2)
+    meta = RunMeta.load(run_dir / "run_meta.json")
+    _fake_venv(tmp_path)
+    cfg = SubmitConfig(run_dir=run_dir, accounting_group="cms", repo_dir=tmp_path, n_chunks=2)
+    write_submit(cfg)
+    jobs = {(i, int(k)): int(w) for i, k, w in (line.split(",") for line in (run_dir / "jobs.txt").read_text().split())}
+    for chunk in range(2):
+        expected = estimate_runtime_s("marginal_edep", meta.rows_per_chunk[chunk])
+        assert jobs[("marginal_edep", chunk)] == expected

tests/test_loader.py

@@ -108,12 +108,12 @@ def test_build_process_map_from_files_spans_multiple_files(tmp_path):
 
 def test_build_process_map_from_files_tie_breaking_pins_first_seen_order(tmp_path):
     """When two processes end up with equal total counts, ranking falls back
-    to whichever was accumulated first (`sorted(..., reverse=True)` is stable,
-    and `counts` is built in file/row-scan order) — this is implementation-
-    defined, not a documented contract, so pin it explicitly: a future
-    rewrite (e.g. a polars-based single-scan) that ties differently would
-    silently reshuffle which processes get their own expert slot across a
-    retrain, and this test is what should catch that."""
+    to whichever was scanned first — file order, then row order within a
+    file (`giant.data.scan`'s `first_seen` ordinal, ranked by
+    `giant.data.loader._topn_plus_other_map`'s `(-count, first_seen)` key).
+    This is an explicit, documented contract (not an accident of iteration
+    order), pinned here so a future change to the ranking can't silently
+    reshuffle which processes get their own expert slot across a retrain."""
     path = tmp_path / "a.parquet"
     pd.DataFrame({"process": ["compt", "phot", "compt", "phot"]}).to_parquet(path)
 
@@ -233,6 +233,23 @@ def test_build_pdg_topn_map_from_files_pools_primary_and_secondary_pdg(tmp_path)
     assert m.class_counts == {0: 11, 1: 5}
 
 
+def test_build_pdg_topn_map_from_files_pooled_tie_breaks_by_row_position(tmp_path):
+    """Pooled pdg counting merges the primary `pdg` column and the exploded
+    `sec_pdg_list` column via one `group_by` over both (see
+    `giant.data.scan._pooled_pdg_lazy`), keyed by row position regardless of
+    which role (primary or secondary) a code was seen in — not "all
+    primaries before all secondaries" the way a two-pass accumulation would.
+    11 (primary, row 0), 33 (primary, row 1) and 22 (secondary, row 1) all
+    end up with count 1; 11's strictly earlier row wins the tie over both,
+    whatever order 33/22 (tied with each other, same row) land in."""
+    path = tmp_path / "a.parquet"
+    pd.DataFrame({"pdg": [11, 33], "sec_pdg_list": [[], [22]]}).to_parquet(path)
+
+    m = build_pdg_topn_map_from_files([path], n_classes=4)
+
+    assert m.class_map[11] == 0
+
+
 def test_build_pdg_topn_map_from_files_missing_sec_pdg_list_column(tmp_path):
     """Files predating the parent->child join have no sec_pdg_list column —
     must not raise, just count the primary pdg column alone."""

tests/test_pipeline.py

@@ -142,7 +142,7 @@ def test_run_train_job_second_run_hits_cache(tmp_path, data, monkeypatch):
     def _forbidden(*a, **k):
         raise AssertionError("should be served from cache, not recomputed")
 
-    monkeypatch.setattr("giant.pipeline.build_index_maps_from_files", _forbidden)
+    monkeypatch.setattr("giant.pipeline.scan_metadata", _forbidden)
     monkeypatch.setattr("giant.pipeline.iter_file_chunks", _forbidden)
 
     echo2 = _run(data, tmp_path / "out2")
@@ -283,7 +283,7 @@ def test_run_train_job_new_val_fraction_is_partial_hit(tmp_path, data, monkeypat
     def _forbidden(*a, **k):
         raise AssertionError("vocab should be served from cache")
 
-    monkeypatch.setattr("giant.pipeline.build_index_maps_from_files", _forbidden)
+    monkeypatch.setattr("giant.pipeline.scan_metadata", _forbidden)
 
     echo2 = _run(data, tmp_path / "out2", cfg=_tiny_cfg(val_fraction=0.3))
     joined = "\n".join(echo2)

tests/test_render.py

@@ -206,21 +206,21 @@ def test_render_all_run_gallery_invokes_subprocess(tmp_path: Path, monkeypatch):
     assert kwargs == {"check": True}
 
 
-def test_render_run_glues_run_meta_into_render_all(tmp_path: Path, monkeypatch):
-    from giant.analysis import run as run_mod
+def test_render_run_glues_condor_run_meta_into_render_all(tmp_path: Path, monkeypatch):
+    from giant.analysis import condor as condor_mod
 
     run_dir = tmp_path / "run"
     (run_dir / "reduced").mkdir(parents=True)
 
     merge_calls = []
-    monkeypatch.setattr(run_mod, "merge_all", lambda rd: merge_calls.append(Path(rd)))
-    meta = run_mod.RunMeta(
+    monkeypatch.setattr(condor_mod, "merge_all", lambda rd: merge_calls.append(Path(rd)))
+    meta = condor_mod.RunMeta(
         rollouts=[{"name": "rollout", "path": "rollout.parquet", "plot_meta": {"checkpoint": "ckpt/best.pt"}}],
         reference="reference.parquet",
         run_dir=str(run_dir),
         title="my-run",
     )
-    monkeypatch.setattr(run_mod.RunMeta, "load", classmethod(lambda cls, p: meta))
+    monkeypatch.setattr(condor_mod.RunMeta, "load", classmethod(lambda cls, p: meta))
 
     Reduced("s", "species", "single_hist", "Single", "x", {"edges": [0, 1], "series": {"rollout": [1]}}).save(
         run_dir / "reduced" / "s.json"

tests/test_workflow_spec.py

@@ -1,148 +0,0 @@
-"""Workflow spec parsing, validation, and spec hashes (gitea #83)."""
-
-import pytest
-
-from giant.workflow.spec import (
-    WorkflowSpecError,
-    epoch_milestones,
-    load_spec,
-    parse_spec,
-    spec_hash,
-)
-
-MINIMAL = {
-    "workflow": {"name": "wf", "result_dir": "/tmp/wf"},
-    "condor": {"accounting_group": "cms", "repo_dir": "/work/lbogner/giant"},
-    "dataset": {"steps": "/data/train", "reference": "/data/holdout"},
-    "train": [{"name": "a", "epochs": 3}],
-    "rollout": [{"name": "a", "train": "a"}],
-    "analysis": [{"name": "cmp", "rollouts": ["a"], "chunks": 4}],
-}
-
-
-def _spec(**patch):
-    raw = {k: (v.copy() if isinstance(v, dict) else list(v)) for k, v in MINIMAL.items()}
-    raw.update(patch)
-    return parse_spec(raw)
-
-
-def test_parses_minimal_spec():
-    spec = _spec()
-    assert spec.name == "wf"
-    assert spec.log_dir == "/tmp/wf/logs"  # derived from result_dir
-    assert spec.train("a").epochs == 3
-    assert spec.rollout("a").train == "a"
-    assert spec.analysis("cmp").rollouts == ("a",)
-    # defaults come from the dataclasses, not the file
-    assert spec.geometry.method == "slab"
-    assert spec.condor.docker_image_gpu == "mschnepf/slc7-condocker"
-
-
-def test_example_config_is_valid():
-    spec = load_spec("configs/workflow_example.toml")
-    assert {t.name for t in spec.trains} == {"baseline", "router-balanced"}
-    assert spec.analysis("baseline-vs-router").rollouts == ("baseline", "router-balanced")
-
-
-@pytest.mark.parametrize(
-    "patch, message",
-    [
-        ({"train": [{"name": "a"}, {"name": "a"}]}, "unique"),
-        ({"rollout": [{"name": "r", "train": "nope"}]}, "names no"),
-        ({"analysis": [{"name": "c", "rollouts": ["nope"]}]}, "not defined"),
-        ({"analysis": [{"name": "c", "rollouts": []}]}, "at least one"),
-        ({"analysis": [{"name": "c", "rollouts": ["a"], "chunks": 0}]}, "chunks must be"),
-        ({"train": [{"name": "a", "epochs": 0}]}, "epochs must be"),
-        ({"train": [{"name": "a", "epchs": 3}]}, "unknown key"),
-        ({"geometry": {"methd": "slab"}}, "unknown key"),
-    ],
-)
-def test_validation_errors(patch, message):
-    with pytest.raises(WorkflowSpecError, match=message):
-        _spec(**patch)
-
-
-def test_unknown_top_level_table_rejected():
-    with pytest.raises(WorkflowSpecError, match="unknown top-level"):
-        _spec(nonsense={})
-
-
-def test_missing_required_table_rejected():
-    raw = {k: v for k, v in MINIMAL.items() if k != "dataset"}
-    with pytest.raises(WorkflowSpecError, match=r"missing required \[dataset\]"):
-        parse_spec(raw)
-
-
-def test_unknown_lookup_names_are_explicit():
-    spec = _spec()
-    with pytest.raises(WorkflowSpecError, match="no \\[\\[train\\]\\] named 'zzz'"):
-        spec.train("zzz")
-
-
-def test_hash_is_stable_and_order_independent():
-    a = _spec()
-    b = parse_spec(
-        {
-            "dataset": MINIMAL["dataset"],
-            "condor": MINIMAL["condor"],
-            "workflow": MINIMAL["workflow"],
-            "train": MINIMAL["train"],
-            "rollout": MINIMAL["rollout"],
-            "analysis": MINIMAL["analysis"],
-        }
-    )
-    assert a.train_hash("a") == b.train_hash("a")
-    assert a.analysis_hash("cmp") == b.analysis_hash("cmp")
-    assert len(a.train_hash("a")) == 8
-
-
-def test_hash_changes_with_own_settings():
-    base = _spec()
-    changed = _spec(train=[{"name": "a", "epochs": 4}])
-    assert base.train_hash("a") != changed.train_hash("a")
-
-
-def test_hash_propagates_from_parents():
-    """A dataset change must move every downstream task's directory."""
-    base = _spec()
-    changed = _spec(dataset={"steps": "/data/other", "reference": "/data/holdout"})
-    assert base.train_hash("a") != changed.train_hash("a")
-    assert base.rollout_hash("a") != changed.rollout_hash("a")
-    assert base.analysis_hash("cmp") != changed.analysis_hash("cmp")
-
-    # ... and so must a change to a training the analysis transitively uses.
-    retrained = _spec(train=[{"name": "a", "epochs": 9}])
-    assert retrained.analysis_hash("cmp") != base.analysis_hash("cmp")
-    # while an unrelated knob on the analysis leaves the training alone
-    rebinned = _spec(analysis=[{"name": "cmp", "rollouts": ["a"], "chunks": 4, "bins": 99}])
-    assert rebinned.train_hash("a") == base.train_hash("a")
-    assert rebinned.analysis_hash("cmp") != base.analysis_hash("cmp")
-
-
-def test_warm_cache_hash_ignores_epochs():
-    """Epoch count doesn't change the setup cache, so it must not re-warm it."""
-    base = _spec()
-    longer = _spec(train=[{"name": "a", "epochs": 50}])
-    assert base.warm_cache_hash("a") == longer.warm_cache_hash("a")
-    other_cfg = _spec(train=[{"name": "a", "epochs": 3, "config": "configs/router.toml"}])
-    assert base.warm_cache_hash("a") != other_cfg.warm_cache_hash("a")
-
-
-def test_spec_hash_expands_dataclasses():
-    spec = _spec()
-    assert spec_hash(spec.dataset) == spec_hash(spec.dataset)
-    assert spec_hash(spec.dataset) != spec_hash(spec.geometry)
-
-
-@pytest.mark.parametrize(
-    "epochs, per_job, expected",
-    [
-        (3, 1, [1, 2, 3]),
-        (10, 3, [3, 6, 9, 10]),
-        (9, 3, [3, 6, 9]),
-        (1, 5, [1]),
-    ],
-)
-def test_epoch_milestones(epochs, per_job, expected):
-    spec = _spec(train=[{"name": "a", "epochs": epochs, "epochs_per_job": per_job}])
-    assert epoch_milestones(spec.train("a")) == expected

tests/test_workflow_tasks.py

@@ -1,172 +0,0 @@
-"""Workflow task graph: dependencies, output paths, condor settings (gitea #83)."""
-
-import pytest
-
-from giant.analysis.catalog import catalog_ids, get_spec as get_plot_spec
-from giant.workflow import tasks
-from giant.workflow.spec import parse_spec
-
-CONDOR = {
-    "accounting_group": "cms",
-    "repo_dir": "/work/lbogner/giant",
-    "env_script": "/work/lbogner/giant/condor_env.sh",
-}
-
-RAW = {
-    "workflow": {"name": "wf", "result_dir": "/results/wf"},
-    "condor": CONDOR,
-    "dataset": {"steps": "/data/train", "reference": "/data/holdout"},
-    "train": [
-        {"name": "base", "epochs": 3, "gpu_memory_mb": 20000},
-        {"name": "router", "epochs": 2},
-    ],
-    "rollout": [
-        {"name": "base", "train": "base"},
-        {"name": "router", "train": "router"},
-    ],
-    "analysis": [{"name": "cmp", "rollouts": ["base", "router"], "chunks": 4}],
-}
-
-
-@pytest.fixture
-def spec():
-    s = parse_spec(RAW)
-    tasks.set_spec(s)
-    return s
-
-
-def _requires(task):
-    return list(task.requires() or [])
-
-
-def test_epoch_chain_is_linear_and_rooted_at_warm_cache(spec):
-    h = spec.train_hash("base")
-    third = tasks.TrainEpochTask(name="base", spec_hash=h, milestone=3)
-    second = _requires(third)
-    assert [type(t) for t in second] == [tasks.TrainEpochTask]
-    assert second[0].milestone == 2
-    first = _requires(second[0])[0]
-    assert first.milestone == 1
-    root = _requires(first)
-    assert [type(t) for t in root] == [tasks.WarmCacheTask]
-    # the warm cache is keyed by its own hash, not the training's
-    assert root[0].spec_hash == spec.warm_cache_hash("base")
-
-
-def test_epoch_task_outputs_last_pt_per_milestone(spec):
-    h = spec.train_hash("base")
-    path = tasks.TrainEpochTask(name="base", spec_hash=h, milestone=2).output().path
-    assert path == f"/results/wf/train_epoch/name=base/spec_hash={h}/epochs=2/last.pt"
-
-
-def test_train_task_requires_final_epoch_and_publishes_canonical_outputs(spec):
-    h = spec.train_hash("base")
-    train = tasks.TrainTask(name="base", spec_hash=h)
-    (dep,) = _requires(train)
-    assert isinstance(dep, tasks.TrainEpochTask) and dep.milestone == 3
-    out = train.output()
-    assert set(out) == {"best.pt", "last.pt", "metrics.csv"}
-    assert out["best.pt"].path == f"/results/wf/train/name=base/spec_hash={h}/best.pt"
-    # local: it only copies files around, no reason to queue a job for it
-    assert train.batch_system == "local"
-
-
-def test_rollout_requires_training_geometry_and_reference(spec):
-    ro = tasks.RolloutTask(name="base", spec_hash=spec.rollout_hash("base"))
-    deps = _requires(ro)
-    assert [type(d) for d in deps] == [tasks.TrainTask, tasks.GeometryOracleTask, tasks.DatasetTask]
-    assert deps[0].name == "base"
-    assert deps[2].path == "/data/holdout"
-    out = ro.output()
-    assert out["rollout.yaml"].path.endswith("rollout.yaml")
-    # the sidecar sits next to the parquet — the deterministic path
-    # `giant rollout --out` now produces
-    assert out["rollout.yaml"].path[: -len(".yaml")] == out["rollout.parquet"].path[: -len(".parquet")]
-
-
-def test_analysis_prep_requires_every_named_rollout(spec):
-    prep = tasks.AnalysisPrepTask(name="cmp", spec_hash=spec.analysis_hash("cmp"))
-    deps = _requires(prep)
-    assert [d.name for d in deps] == ["base", "router"]
-    assert all(isinstance(d, tasks.RolloutTask) for d in deps)
-    assert prep.batch_system == "local"
-
-
-def test_compute_job_enumeration_collapses_non_chunkable_specs(spec):
-    jobs = tasks.analysis_jobs(spec, "cmp")
-    non_chunkable = [i for i in catalog_ids() if not get_plot_spec(i).chunkable]
-    expected = (len(catalog_ids()) - len(non_chunkable)) * 4 + len(non_chunkable)
-    assert len(jobs) == expected
-    assert non_chunkable, "expected some chunkable=False specs in the catalog"
-    for spec_id in non_chunkable:
-        assert [c for i, c in jobs if i == spec_id] == [0]
-
-
-def test_compute_output_matches_the_on_disk_contract(spec):
-    h = spec.analysis_hash("cmp")
-    task = tasks.AnalysisComputeTask(name="cmp", spec_hash=h, plot_id="event_mean_length", chunk=2)
-    assert task.output().path == (
-        f"/results/wf/analysis/name=cmp/spec_hash={h}/reduced_partial/event_mean_length__2.json"
-    )
-    (dep,) = _requires(task)
-    assert isinstance(dep, tasks.AnalysisPrepTask)
-
-
-def test_render_requires_every_compute_job_and_runs_locally(spec):
-    render = tasks.AnalysisRenderTask(name="cmp", spec_hash=spec.analysis_hash("cmp"))
-    deps = _requires(render)
-    assert len(deps) == len(tasks.analysis_jobs(spec, "cmp"))
-    assert render.batch_system == "local"  # the only step importing plotstyle/LaTeX
-    assert render.output().path.endswith("/plots/metadata.yaml")
-
-
-def test_workflow_task_wraps_every_analysis(spec):
-    deps = _requires(tasks.WorkflowTask(workflow_name="wf"))
-    assert [(type(d), d.name) for d in deps] == [(tasks.AnalysisRenderTask, "cmp")]
-
-
-def test_workflow_without_analysis_falls_back_to_rollouts():
-    raw = {k: v for k, v in RAW.items() if k != "analysis"}
-    tasks.set_spec(parse_spec(raw))
-    deps = _requires(tasks.WorkflowTask(workflow_name="wf"))
-    assert [type(d) for d in deps] == [tasks.RolloutTask, tasks.RolloutTask]
-
-
-def test_gpu_settings_carry_remote_ceph_and_pins(spec):
-    settings = tasks.TrainEpochTask(name="base", spec_hash=spec.train_hash("base"), milestone=1).htcondor_settings
-    assert settings["+RemoteJob"] == "True"
-    assert settings["RequestGPUs"] == 1
-    assert "TARGET.ProvidesEtpCeph =?= True" in settings["requirements"]
-    assert "TARGET.GPUs_GlobalMemoryMb >= 20000" in settings["requirements"]
-    assert settings["accounting_group"] == "cms"
-    assert settings["docker_image"] == "mschnepf/slc7-condocker"
-
-
-def test_cpu_settings_used_for_analysis_compute(spec):
-    task = tasks.AnalysisComputeTask(
-        name="cmp", spec_hash=spec.analysis_hash("cmp"), plot_id="event_mean_length", chunk=0
-    )
-    settings = task.htcondor_settings
-    assert settings["docker_image"] == "cverstege/alma9-gridjob"
-    assert "RequestGPUs" not in settings
-    # no run_meta.json yet (prep hasn't run), so no walltime is claimed
-    assert "+RequestWalltime" not in settings
-
-
-def test_cpu_settings_local_files_use_provides_etp_resources():
-    raw = {**RAW, "condor": {**CONDOR, "remote": False}}
-    spec = parse_spec(raw)
-    tasks.set_spec(spec)
-    settings = tasks.GeometryOracleTask(spec_hash=spec.geometry_hash()).htcondor_settings
-    assert settings["requirements"] == "TARGET.ProvidesETPResources"
-    assert "+RemoteJob" not in settings
-
-
-def test_missing_dataset_fails_immediately(spec):
-    with pytest.raises(FileNotFoundError, match="/ceph"):
-        tasks.DatasetTask(path="/data/train").complete()
-
-
-def test_dataset_that_exists_is_complete(tmp_path, spec):
-    (tmp_path / "steps.parquet").write_text("")
-    assert tasks.DatasetTask(path=str(tmp_path / "steps.parquet")).complete()

uv.lock

@@ -147,26 +147,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/44/a1/70ebfffd6c6edc6034a547838ee46287c65ed89f710592ddc39c76b4a5a8/awkward_cpp-53-cp314-cp314t-win_arm64.whl", hash = "sha256:1be0c1d87d9f4fdf94b767a061df849f1bb21579d302b2996fb101527fc80a97", size = 551257, upload-time = "2026-06-08T12:31:56.319Z" },
 ]
 
-[[package]]
-name = "b2luigi"
-version = "1.2.9"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "cachetools" },
-    { name = "colorama" },
-    { name = "gitpython" },
-    { name = "jinja2" },
-    { name = "luigi" },
-    { name = "parse" },
-    { name = "setuptools" },
-    { name = "tenacity" },
-    { name = "webdavclient3" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/e5/5d/0c3e0602b6cf80a2cfebbe54c330f3623b4de227c2b0f8cbff437dc8d62c/b2luigi-1.2.9.tar.gz", hash = "sha256:3f6734b06970cd5bf6bb62c45c095e9c02b6e283a45b76a29ffabdeaa8fce0c0", size = 786252, upload-time = "2026-04-17T16:38:23.316Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/c3/82/1a8d3c20bc235e665300c42783eb8bf9f2aac49c7eae644273191e5e01e8/b2luigi-1.2.9-py3-none-any.whl", hash = "sha256:9ca28b4203f5946394b609da432e53b3c9e35540c840bdc6497cce6ab4c82e2e", size = 115103, upload-time = "2026-04-17T16:38:21.044Z" },
-]
-
 [[package]]
 name = "bracex"
 version = "3.0.1"
@@ -196,15 +176,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/b3/0b/5885530f79d4400368b9d4dcb9b39274c0d52e633f7871e7fc6feceea1e3/bump_my_version-1.5.1-py3-none-any.whl", hash = "sha256:df3e2989d0d7fe704718feb24a5880f089b6b6369e427a4445b89c3adebfcff1", size = 65090, upload-time = "2026-08-06T14:26:37.083Z" },
 ]
 
-[[package]]
-name = "cachetools"
-version = "7.1.7"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/70/d2/47e8bc06fe2a06d3f5bdf20f1126ab66c4e99dc48d940e7ba873f7ac7131/cachetools-7.1.7.tar.gz", hash = "sha256:a3e2a00b14d8f8a6b70c1dae7b4685e7ad3bc965c5b42124a2d6ce895da6cf50", size = 40680, upload-time = "2026-08-01T21:20:40.434Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/e4/d8/767faeda872075724b95dd675466a645f1b92aadcdcf2d1429dcfd76c176/cachetools-7.1.7-py3-none-any.whl", hash = "sha256:ef98ef375ad188819ef2f9b3645e3987f4b8c5b7550e436ad998c2de78296df0", size = 16830, upload-time = "2026-08-01T21:20:38.977Z" },
-]
-
 [[package]]
 name = "certifi"
 version = "2026.7.22"
@@ -634,15 +605,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/05/7f/798705f5296a58ca505d600456748d1be48078eac8a7050d8a98bc9edb89/decorator-5.3.1-py3-none-any.whl", hash = "sha256:f47fe6fdbd2edd623ecfe36875d37aba411624e2670dd395dddae1358689bb3c", size = 10365, upload-time = "2026-05-18T06:03:26.517Z" },
 ]
 
-[[package]]
-name = "docutils"
-version = "0.23"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/39/a4/5180d9afc57e8fca05601dd652bdff19604c218814037fe90ffc7625a50a/docutils-0.23.tar.gz", hash = "sha256:746f5060322511280a1e50eb76846ed6bf2342984b2ac04dc42caa1a8d78799e", size = 2303823, upload-time = "2026-05-27T17:41:06.934Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/32/91/30151a39f7570f448ed84529390628a651d7f27c87d73c9b887f8189695e/docutils-0.23-py3-none-any.whl", hash = "sha256:25d013af9bf23bc1c7b2b093dff4208166c53a94786c9e447808335ef1185fea", size = 634701, upload-time = "2026-05-27T17:40:58.442Z" },
-]
-
 [[package]]
 name = "executing"
 version = "2.2.1"
@@ -713,12 +675,12 @@ wheels = [
 
 [[package]]
 name = "giant"
-version = "0.4.0"
+version = "0.3.16"
 source = { editable = "." }
 dependencies = [
     { name = "numpy" },
-    { name = "pandas" },
     { name = "particle" },
+    { name = "polars" },
     { name = "pyarrow" },
     { name = "pyyaml" },
     { name = "tqdm" },
@@ -746,11 +708,11 @@ cuda = [
 ]
 dev = [
     { name = "awkward" },
-    { name = "b2luigi" },
     { name = "bump-my-version" },
     { name = "git-cliff" },
     { name = "ipykernel" },
     { name = "matplotlib" },
+    { name = "pandas" },
     { name = "plotstyle" },
     { name = "polars" },
     { name = "pytest" },
@@ -767,23 +729,20 @@ geometry = [
 wandb = [
     { name = "wandb" },
 ]
-workflow = [
-    { name = "b2luigi" },
-]
 
 [package.metadata]
 requires-dist = [
     { name = "awkward", marker = "extra == 'convert'", specifier = ">=2.6,<3" },
-    { name = "b2luigi", marker = "extra == 'workflow'", specifier = ">=1.0,<2" },
     { name = "bump-my-version", marker = "extra == 'dev'", specifier = ">=1.2,<2" },
-    { name = "giant", extras = ["convert", "analysis", "geometry", "wandb", "workflow"], marker = "extra == 'dev'" },
+    { name = "giant", extras = ["convert", "analysis", "geometry", "wandb"], marker = "extra == 'dev'" },
     { name = "git-cliff", marker = "extra == 'dev'", specifier = ">=2,<3" },
     { name = "ipykernel", marker = "extra == 'analysis'", specifier = ">=7.3.0" },
     { name = "matplotlib", marker = "extra == 'analysis'", specifier = ">=3.8,<4" },
     { name = "numpy", specifier = ">=1.26,<3" },
-    { name = "pandas", specifier = ">=2.2,<4" },
+    { name = "pandas", marker = "extra == 'dev'", specifier = ">=2.2,<4" },
     { name = "particle", specifier = ">=1.0,<2" },
     { name = "plotstyle", marker = "extra == 'analysis'", specifier = ">=1.0.0", index = "https://git.larsbogner.de/api/packages/lars/pypi/simple/" },
+    { name = "polars", specifier = ">=1.0,<2" },
     { name = "polars", marker = "extra == 'analysis'", specifier = ">=1.0,<2" },
     { name = "polars", marker = "extra == 'convert'", specifier = ">=1.0,<2" },
     { name = "pyarrow", specifier = ">=16,<25" },
@@ -800,7 +759,7 @@ requires-dist = [
     { name = "uproot", marker = "extra == 'convert'", specifier = ">=5.3,<6" },
     { name = "wandb", marker = "extra == 'wandb'", specifier = ">=0.16,<1" },
 ]
-provides-extras = ["cpu", "cuda", "dev", "geometry", "wandb", "convert", "analysis", "workflow"]
+provides-extras = ["cpu", "cuda", "dev", "geometry", "wandb", "convert", "analysis"]
 
 [[package]]
 name = "git-cliff"
@@ -822,30 +781,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/99/b2/99fac50978b9a90bfec0f1b89354a667ec83f4990301f6c708abce05484e/git_cliff-2.13.1-py3-none-win_amd64.whl", hash = "sha256:856d831a0bede9c258229dbd4d4c2b1c0810d8fce3d3882729669e8dc09c72bf", size = 7714969, upload-time = "2026-04-26T10:33:40.163Z" },
 ]
 
-[[package]]
-name = "gitdb"
-version = "4.0.12"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "smmap" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/72/94/63b0fc47eb32792c7ba1fe1b694daec9a63620db1e313033d18140c2320a/gitdb-4.0.12.tar.gz", hash = "sha256:5ef71f855d191a3326fcfbc0d5da835f26b13fbcba60c32c21091c349ffdb571", size = 394684, upload-time = "2025-01-02T07:20:46.413Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/a0/61/5c78b91c3143ed5c14207f463aecfc8f9dbb5092fb2869baf37c273b2705/gitdb-4.0.12-py3-none-any.whl", hash = "sha256:67073e15955400952c6565cc3e707c554a4eea2e428946f7a4c162fab9bd9bcf", size = 62794, upload-time = "2025-01-02T07:20:43.624Z" },
-]
-
-[[package]]
-name = "gitpython"
-version = "3.1.60"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "gitdb" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/84/14/e6b1a48d831755a53c2029351fcef82e70db4a08f338daefe29d8d0cf31c/gitpython-3.1.60.tar.gz", hash = "sha256:e936431879fa85581b4311fa63492ea52251909e2d655b6529c704c904ddcc24", size = 230793, upload-time = "2026-08-25T18:33:46.102Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/71/63/ba28697918b7c190af9f3f21940d03e8814e25dd4ddd39d6929f3a553995/gitpython-3.1.60-py3-none-any.whl", hash = "sha256:39548bffb8fa0f3a548133348868bb4838e79d73283052207dc97781a569b6b4", size = 221893, upload-time = "2026-08-25T18:33:44.75Z" },
-]
-
 [[package]]
 name = "h11"
 version = "0.16.0"
@@ -1140,141 +1075,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/b5/91/53255615acd2a1eaca307ede3c90eb550bae9c94581f8c00081b6b1c8f44/kiwisolver-1.5.0-graalpy312-graalpy250_312_native-win_amd64.whl", hash = "sha256:1f1489f769582498610e015a8ef2d36f28f505ab3096d0e16b4858a9ec214f57", size = 75987, upload-time = "2026-03-09T13:15:39.65Z" },
 ]
 
-[[package]]
-name = "lockfile"
-version = "0.12.2"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/17/47/72cb04a58a35ec495f96984dddb48232b551aafb95bde614605b754fe6f7/lockfile-0.12.2.tar.gz", hash = "sha256:6aed02de03cba24efabcd600b30540140634fc06cfa603822d508d5361e9f799", size = 20874, upload-time = "2015-11-25T18:29:58.279Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/c8/22/9460e311f340cb62d26a38c419b1381b8593b0bb6b5d1f056938b086d362/lockfile-0.12.2-py2.py3-none-any.whl", hash = "sha256:6c3cb24f344923d30b2785d5ad75182c8ea7ac1b6171b08657258ec7429d50fa", size = 13564, upload-time = "2015-11-25T18:29:51.462Z" },
-]
-
-[[package]]
-name = "luigi"
-version = "3.7.3"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "python-daemon" },
-    { name = "python-dateutil" },
-    { name = "tenacity" },
-    { name = "tornado" },
-    { name = "typing-extensions" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/b0/df/5235521bd30402e19e469b79c9e91757046014a3d9bc2c91fd341b0edcb6/luigi-3.7.3.tar.gz", hash = "sha256:82e3a5d482e9738f4dcae5acb631ee7ffa748896cb1a1d5d4922c2cac61903fe", size = 1295810, upload-time = "2026-02-12T01:49:34.147Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/63/42/a311f89099c5714dc9c69368d4197a99d453d5f3764713eb5c09b8a8cf18/luigi-3.7.3-py3-none-any.whl", hash = "sha256:f6aaac2417e28a833d30e6b1cd03fe53b2254024dd861fa104ab342cc4f3b843", size = 1186711, upload-time = "2026-02-12T01:49:32.78Z" },
-]
-
-[[package]]
-name = "lxml"
-version = "6.1.2"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/ad/a9/970b8fa0ecc4fbf1dfaed0d89bbc1fc1421b25ec26a2038c91e872dc6c8e/lxml-6.1.2.tar.gz", hash = "sha256:1055241852f2b02068af4a625a5d32c087db193c12251928af2562ecd2239f18", size = 4210626, upload-time = "2026-08-19T04:58:15.341Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/ee/a4/55eb54507073089ab27743c5da2113c84f0d0b1715b33175fdd943c9652d/lxml-6.1.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:7d506bdba580ecb1a6ad2e2b5c49445e66d3e1f95894885739094393a1aad237", size = 8602111, upload-time = "2026-08-19T04:58:28.017Z" },
-    { url = "https://files.pythonhosted.org/packages/bc/bf/6332f45d78da385bb01d5cac3fe4acda19f025d1307cbc7ad538355fecbb/lxml-6.1.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:12acd337d2821cb8b9247dfe4b7aa2f2769a3df5ae8511b7e550df42b8f4d3c3", size = 4638376, upload-time = "2026-08-19T04:58:41.181Z" },
-    { url = "https://files.pythonhosted.org/packages/68/e0/21fba0fe74d417fbe976903ae6bc77e92cdce01aae7b636abd87756f4588/lxml-6.1.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5078ff51e6316c0f75ea8127c2cd24374747fb351f62fb93d1761f8ae5a04a40", size = 4939689, upload-time = "2026-08-19T04:58:48.526Z" },
-    { url = "https://files.pythonhosted.org/packages/de/e5/ce3e885264fdd0bdcb6b49c1ea1842f94281b39e4ff956099e8d57532c60/lxml-6.1.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9477e14217c212e6023c994a71a1a349db19b0e10fd5bf189666b281ae63b1fd", size = 5105185, upload-time = "2026-08-19T04:59:15.533Z" },
-    { url = "https://files.pythonhosted.org/packages/e6/b6/990a8446c488c70fa25681e150de94b7bf2eaaf387e374d195ab3c8faafb/lxml-6.1.2-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:261d98065326676d7253882db0198d0aa06748d7ee0443367acf10b148273f99", size = 5011863, upload-time = "2026-08-19T04:59:50.58Z" },
-    { url = "https://files.pythonhosted.org/packages/bb/6a/f70f41363dae27e3bfd6224b128f5ba150874bd32ca4938552930ffa33b0/lxml-6.1.2-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:0666943ee1576fa890a6dc6316ef42e8241b5dd56f67bc5475acb2ac298c6ca9", size = 5638234, upload-time = "2026-08-19T05:00:00.802Z" },
-    { url = "https://files.pythonhosted.org/packages/3d/e0/a65b64f34d556925faef2c4f14167d58c571bc15a3e1f2bba71138830562/lxml-6.1.2-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:04cf9e3f4ee9cab9d9ba05401bef8668840fa9620fcd4d8e85a2d2fd0b0fa960", size = 5244532, upload-time = "2026-08-19T05:00:07.516Z" },
-    { url = "https://files.pythonhosted.org/packages/c6/a9/471552e015e954fc9d960aa27c3d67ebf489683d03f033399a790417c67c/lxml-6.1.2-cp312-cp312-manylinux_2_28_i686.whl", hash = "sha256:9429d2371d406344ed1da5b5686d9412e74137c07b0171278368ff704f470ed5", size = 5358194, upload-time = "2026-08-19T05:00:22.747Z" },
-    { url = "https://files.pythonhosted.org/packages/d7/0f/bc6248fbec2cc416f102b1267f1567e07510f6fa909bbe8cd2a22d6fb78e/lxml-6.1.2-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:eff128ffdc093cc6317955934ad9751105d37ed8dbca3ff4ccd751af6be37185", size = 4704432, upload-time = "2026-08-19T05:00:51.115Z" },
-    { url = "https://files.pythonhosted.org/packages/a9/3f/cec859f50e63f1fa338fab43d2362d7543e1237f2475960d8ab0769de0eb/lxml-6.1.2-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ba58574d710b82ead7cbedea01cac3e110bc3ef82d4731519b74a2c11f7cf5e9", size = 5255038, upload-time = "2026-08-19T05:00:58.895Z" },
-    { url = "https://files.pythonhosted.org/packages/7c/d9/2ced0cf2967115f92a1b8b3ae6bd18763abc3ebef88c98cf25145fda396c/lxml-6.1.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:52f6d4dff133c9778a24e9a2cfc1608930b15869866171aacc5131b5a418a003", size = 5054481, upload-time = "2026-08-19T05:01:10.096Z" },
-    { url = "https://files.pythonhosted.org/packages/d5/f5/4f07386d3c88673daeec3b8cc09a2a4d39fa01c1fc49009791b0746d97fa/lxml-6.1.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:8807998c1023d1e9d60e02500f90e85a0752dbc0b670989806bba87b82dd5b42", size = 4785535, upload-time = "2026-08-19T05:01:18.909Z" },
-    { url = "https://files.pythonhosted.org/packages/9a/5a/f4fe3ecbc189f48fba2547c5db5c940a10151d3e86b856a60a533a77e816/lxml-6.1.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:2170d0a280c877b6e2dc6738217db947be35dd8cf09ca458b355aa1bab2a9e70", size = 5655337, upload-time = "2026-08-19T05:01:41.324Z" },
-    { url = "https://files.pythonhosted.org/packages/92/c4/f586aa1bf27bfbace2dfdbb704da5c52f0bdece8ee440c8fb4946c940b2e/lxml-6.1.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:c67f3c1278f942e97d8665c2a690324aaea5137de16f056583a21f0ac706177f", size = 5245778, upload-time = "2026-08-19T05:01:45.227Z" },
-    { url = "https://files.pythonhosted.org/packages/18/a1/677494bbaef4d6db5e4633af817414f478865850b55c03ae4bf70fa7b8ca/lxml-6.1.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:093fbf547d0f3ca02705381f795a050fbb58988be4aac7f79f99f280c4082313", size = 5267274, upload-time = "2026-08-19T05:01:57.687Z" },
-    { url = "https://files.pythonhosted.org/packages/5a/71/b71425b8764d4cb7c92eb970483be7d5610dce2a6316242b5aaae7d260be/lxml-6.1.2-cp312-cp312-win32.whl", hash = "sha256:be365ce8d2d411cf2fb573747684b4fd470fa6224e0094d9d5a21155acc369d3", size = 3602563, upload-time = "2026-08-19T05:02:01.837Z" },
-    { url = "https://files.pythonhosted.org/packages/1b/fb/909584e16d2148c1a252cc2c32dd99fe0e2682459c586d3d7a192e74a0ae/lxml-6.1.2-cp312-cp312-win_amd64.whl", hash = "sha256:b97153ca609b434b712ddfb92cd6af101a7045a7724c542258bd4727a344472f", size = 4005965, upload-time = "2026-08-19T05:02:07.157Z" },
-    { url = "https://files.pythonhosted.org/packages/5f/8d/41207c9212caad0b52749e34739fb9bfab67486729f52a8fe9bd9266fee6/lxml-6.1.2-cp312-cp312-win_arm64.whl", hash = "sha256:7feb72424f19a893ae4f3373c7aae821b1aacb6076b708915c651f0683a97c49", size = 3666641, upload-time = "2026-08-19T05:02:11.3Z" },
-    { url = "https://files.pythonhosted.org/packages/61/2a/e9651f47a31a60b5cae031abc23391ed9aa30c8fc07571d1a38f58d6d770/lxml-6.1.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:351318f5c0eb7fcab5b4fdb507c6f88fb2c4b5e67784c7e5911448c91fffb5d4", size = 8590165, upload-time = "2026-08-19T04:58:40.489Z" },
-    { url = "https://files.pythonhosted.org/packages/61/87/a8098abaf35118767d1703b84c98940a5d833064e0eca39a00ecfe9840ab/lxml-6.1.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:c0edde95e4b4278dcc0175eda06dc8aa2631ad9f83ae5dbdbc4f0925e200b0b0", size = 4632474, upload-time = "2026-08-19T04:58:47.465Z" },
-    { url = "https://files.pythonhosted.org/packages/93/cc/fe74d1def7f4fb967c4a825608a074d4dbdbb871b0d6bd59c6ed07d67868/lxml-6.1.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a8326e24ae6c3a6bfb03fa8b4793f9a5d804c125228aa067f652b0428e31b87c", size = 4936196, upload-time = "2026-08-19T04:59:03.477Z" },
-    { url = "https://files.pythonhosted.org/packages/b7/ad/b96e6ca926e26726a99aa643602aac7411ecc1731ddb1b25af8cc57edfcd/lxml-6.1.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7c534ed898413f439b048130011e99a4245ee13d62d431f6b4f7f2484d02a93a", size = 5093290, upload-time = "2026-08-19T04:59:17.498Z" },
-    { url = "https://files.pythonhosted.org/packages/d2/84/616f5d3b7cd086fcfba3e5add6fccda67f976c1c753ae9ed7bbd317cb9be/lxml-6.1.2-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2e37fe49fe2d5aa40a2cb1cc8176673ad7de0d124e6f4a509d9318f5979c7871", size = 4998767, upload-time = "2026-08-19T04:59:28.385Z" },
-    { url = "https://files.pythonhosted.org/packages/80/88/d5b453a8d083483c9442ad7f5ac5c560796022eb5c80d60b65d75e449236/lxml-6.1.2-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9b52ea73a37fc64aa3357ff8607801d46dd170506d3cf8253a91a1d91639d4f9", size = 5626717, upload-time = "2026-08-19T04:59:40.045Z" },
-    { url = "https://files.pythonhosted.org/packages/71/45/31e5aa4d4bae024908ba1d03480c7425cf027a28b7e5c88d1b7202bd80cc/lxml-6.1.2-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e8b9a92652e75e7731309ea51db5dee892eef414ce70a6ec3441e5d36bf5189f", size = 5232330, upload-time = "2026-08-19T04:59:46.175Z" },
-    { url = "https://files.pythonhosted.org/packages/f5/5a/2627912420df8b2d31ba3014da5539f15ec85add01d42048864ffefda516/lxml-6.1.2-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:9088da25ecd609965f838d89fda0465a905b48f4dd90331db9845518f2177372", size = 5347054, upload-time = "2026-08-19T04:59:52.762Z" },
-    { url = "https://files.pythonhosted.org/packages/16/86/54ac0f529b22a8f12313726dd49e12961bb46471d9028cc28d2a29408f0b/lxml-6.1.2-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:0349321a0537d4fdbebb2af06dd1b64676132c72e2ae250de8cdb58f8c43019c", size = 4707275, upload-time = "2026-08-19T05:00:04.836Z" },
-    { url = "https://files.pythonhosted.org/packages/3a/42/ffcdc6e4519be90df907cdae7e88409efb25d823ae4de8846f737dae1884/lxml-6.1.2-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b20440e578d269c5e8a722ab602ddd0f0cedb8b080006b3f936da9991a593d3b", size = 5240071, upload-time = "2026-08-19T05:00:19.604Z" },
-    { url = "https://files.pythonhosted.org/packages/68/49/5b1d7ab35f013f1127ec48f3108319f58b65b00d5cb26f215adbe86eadfb/lxml-6.1.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:7766e525282dd38fd89567311323e441996eb958e8e816d16b38f782e3aecd2a", size = 5050356, upload-time = "2026-08-19T05:00:27.968Z" },
-    { url = "https://files.pythonhosted.org/packages/b0/57/1cf049d054189b55c8fe8012269234f6602256949b69cd3ba80608a88219/lxml-6.1.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:9221442682c27417f10fe11184ea4cce174b25ab52465570b1f3ee3f85f320fa", size = 4780394, upload-time = "2026-08-19T05:00:39.047Z" },
-    { url = "https://files.pythonhosted.org/packages/d5/ad/064488a8fa60e639fd773e421a18bf17541d02a95fbf36238ad7c65f69d4/lxml-6.1.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:75530642d8471327e691ab9b0513a5f9c77f38871014ceda40f51bb51765c0a1", size = 5645854, upload-time = "2026-08-19T05:03:42.697Z" },
-    { url = "https://files.pythonhosted.org/packages/85/bb/120e56f3cf1c149bb3b014278fb86d0a6dd552403981081f0ee0a0a57be7/lxml-6.1.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:678e35f1cbca98f55107511ee21a60568535c950f3c2371819bd64504c980d20", size = 5231132, upload-time = "2026-08-19T05:03:45.466Z" },
-    { url = "https://files.pythonhosted.org/packages/b1/2c/7d49aab893c128671a3276580074cce4c002896145b8dd2893da79633bca/lxml-6.1.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:5c2bae42b3a09f977330a08f4a8fe72aec58c4bdb89069d3fe7272a71d885881", size = 5256076, upload-time = "2026-08-19T05:03:48.092Z" },
-    { url = "https://files.pythonhosted.org/packages/72/28/ddea3aa1fa9acfd384fe34d4a2a93eecc07541dd2d922fa9b140c60d8014/lxml-6.1.2-cp313-cp313-win32.whl", hash = "sha256:5848f3de6a8de8a93cff9f068134393ff5fa69ac2a04399f7d49cd67c61c348c", size = 3602177, upload-time = "2026-08-19T05:03:50.571Z" },
-    { url = "https://files.pythonhosted.org/packages/1b/7a/96bac167538748cae2544335855f812fa33e49a9a67bc8b8520dcbd592bd/lxml-6.1.2-cp313-cp313-win_amd64.whl", hash = "sha256:6cb0c87421946030b92b558be416852780a912454e3dcba0998e4497c9c588d5", size = 4004117, upload-time = "2026-08-19T05:03:53.074Z" },
-    { url = "https://files.pythonhosted.org/packages/0a/24/9498fa3c84135956e5ef55ea4d8bd11e999e381f7f210fb6f8c6a980ef03/lxml-6.1.2-cp313-cp313-win_arm64.whl", hash = "sha256:648861c19b775b89ebefa14586f85090b10163367476d77f242c4131c835ce73", size = 3665412, upload-time = "2026-08-19T05:03:55.621Z" },
-    { url = "https://files.pythonhosted.org/packages/27/b5/728b0578791b397ace8d1b101c8b3fe10f36043542f7bb85f82d8bdc3f50/lxml-6.1.2-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:d50a44113fe6800dcc8a859332b823a4735b1e6ae1b0063882e4cca569ec3e29", size = 8609651, upload-time = "2026-08-19T04:58:42.42Z" },
-    { url = "https://files.pythonhosted.org/packages/3c/6b/49209fa6225c15c48a30061f03d3aba75e3c19634813b88bf83b88c525ed/lxml-6.1.2-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:fa813b0247d0543a563b993ac3dba6168eef59e3a61448432cf5453300c2412b", size = 4639588, upload-time = "2026-08-19T04:59:01.501Z" },
-    { url = "https://files.pythonhosted.org/packages/20/86/80bae4e8bc2eed9d6f017701a3d86fdea56936218efa738911d0b76aa7f4/lxml-6.1.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d858e718b94033ab4b67e4a58fe3114c65bae01ae2314a62fb39ae8897ed4324", size = 4964846, upload-time = "2026-08-19T04:59:08.59Z" },
-    { url = "https://files.pythonhosted.org/packages/70/ce/4782caee7a22959c1ac67cb46495e03912c22a4ba7d20c163496a519e815/lxml-6.1.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:3e3b666f57a5d81562f38c766c762416b0f6eb58a00590546911514b48412abd", size = 5099288, upload-time = "2026-08-19T04:59:18.649Z" },
-    { url = "https://files.pythonhosted.org/packages/32/21/f120967cc43b54e05512dff0c39726b832c836195d30f41f88733ef36ac8/lxml-6.1.2-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:26ff164c6629e5c4d11c9e55d5ea3d6eed0be2a420eee1f55cbce6e2c23e231a", size = 5036837, upload-time = "2026-08-19T04:59:47.217Z" },
-    { url = "https://files.pythonhosted.org/packages/61/ba/8005e9f47598e3ec5c18312c77f94e889580027616678848405c6aeba5de/lxml-6.1.2-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:962c12b51d0b164f12569af225dea57568477e24a845b96eaccbef6c07e4cc03", size = 5658569, upload-time = "2026-08-19T04:59:54.078Z" },
-    { url = "https://files.pythonhosted.org/packages/6a/ba/add33b3c7ce51462cf7a4637bcfec2eaa258364d6015b989dd7d1216e6a6/lxml-6.1.2-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:47e367dfe341521426692819803e260d0673899c0ff611f14af978d725e2c999", size = 5246003, upload-time = "2026-08-19T04:59:59.764Z" },
-    { url = "https://files.pythonhosted.org/packages/05/b3/a43012748fb861c914c5eac1c1a3bad44282e767499cd02280d4d1edf092/lxml-6.1.2-cp314-cp314-manylinux_2_28_i686.whl", hash = "sha256:92c2b366028ac01e90399e6d17734ce6e4f4aeddd8ba75fbaf80ea11d6c6d645", size = 5354047, upload-time = "2026-08-19T05:00:21.657Z" },
-    { url = "https://files.pythonhosted.org/packages/8b/cb/813021d9a445713b8d758b9e5eae2ed392cd598d9f119d9b053b37c2ab93/lxml-6.1.2-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:7e81fc065ede5d58dd0bf0912025aee1bd04c52c2affd61fdb93226a97ce2fc6", size = 4704382, upload-time = "2026-08-19T05:00:47.067Z" },
-    { url = "https://files.pythonhosted.org/packages/17/c9/1155299f4577bebf3c280497534a73e4b8ad8cab3b96074731ad10949d4e/lxml-6.1.2-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:633ac039cb32366dd5935868e041e385875c017b8cd54ea56aeee3fe29ca5935", size = 5258530, upload-time = "2026-08-19T05:01:14.893Z" },
-    { url = "https://files.pythonhosted.org/packages/25/6e/d76e58384b378b877e140e25b9a9835da00035f81ff70cbe943a3749bf27/lxml-6.1.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:2f3194777c0d05945ac91d8594be25d2679d1d826e01e1fc90bae568ff3a547b", size = 5089919, upload-time = "2026-08-19T05:01:33.602Z" },
-    { url = "https://files.pythonhosted.org/packages/f3/b7/898013c0f8891481d0624ab3bd5dd8c8ff827232dfee2a5d1f8bf970a7cc/lxml-6.1.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:1133bd969f2bfcc6b0c0cf7cdf5f2631e62b23fa2471ee8bd44f6ab73554ee9a", size = 4741972, upload-time = "2026-08-19T05:01:38.18Z" },
-    { url = "https://files.pythonhosted.org/packages/c3/47/efb53c4d7b655831c03317a450d9da439b0829c61f34d9d4fe7c863445d6/lxml-6.1.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:1edca8f4a92b94e873093df959f141d388f2141fcad0c47598442fb4730ef57a", size = 5683241, upload-time = "2026-08-19T05:02:00.731Z" },
-    { url = "https://files.pythonhosted.org/packages/da/0a/0ff36a584cbba14a71326ee8a5300694400f0b97927d1f90a87d95b17d4a/lxml-6.1.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:8512b3775d68994dd1d6d533161e0a214f2ad9c634659d34a99c98e86c6c3d68", size = 5245892, upload-time = "2026-08-19T05:02:06.108Z" },
-    { url = "https://files.pythonhosted.org/packages/8f/9e/303717a1aa56d4bd775c91936717d3c9e8d999a8e8b68b00979c4c1f93d0/lxml-6.1.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a5005c0c9e4d749a76a2ff8bd5918a8bb248df8e08e73a55654b9f79c9cd1e2b", size = 5269528, upload-time = "2026-08-19T05:02:09.883Z" },
-    { url = "https://files.pythonhosted.org/packages/ad/c2/2ae7cb97089eb86bf0689516db3cf280a007b6145853d2a0235a1f01683d/lxml-6.1.2-cp314-cp314-win32.whl", hash = "sha256:e17e2c30e27f56da5551e7a425888b45f013e940b99ab07d125a1c33f77a4605", size = 3662743, upload-time = "2026-08-19T05:03:02.513Z" },
-    { url = "https://files.pythonhosted.org/packages/77/13/a3d483230a09201e211ceb1aa208b1374d27d23b8b180d74dba14b30f6b3/lxml-6.1.2-cp314-cp314-win_amd64.whl", hash = "sha256:87e9673cd8a3445024fe38e7f91b55fa3428437eec9b7a7ff7d81979520c0d2d", size = 4073942, upload-time = "2026-08-19T05:03:04.864Z" },
-    { url = "https://files.pythonhosted.org/packages/1c/f1/c1445d4b6ad7c51e39d4e2ebbf015a4880f5b297a4ab0e77e4d0e5b70110/lxml-6.1.2-cp314-cp314-win_arm64.whl", hash = "sha256:878e7c8ada8f92c52f13f35a2ab98ef0adf7fd0211d164fc2af589e4c3cfed63", size = 3749235, upload-time = "2026-08-19T05:03:07.239Z" },
-    { url = "https://files.pythonhosted.org/packages/9d/eb/598c76f4ce19a67c635e86a46d880cc854f308f39a6f1fdf13bbb01813ec/lxml-6.1.2-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:94162456ed0a64fb1c06915df5bd06af4675ae3966d6048fcb73b0906e0e0222", size = 8860315, upload-time = "2026-08-19T05:02:14.39Z" },
-    { url = "https://files.pythonhosted.org/packages/da/c7/1f9fac7b566a86ad0da13dcc0259164266469c0ad86744c740ccd5c2a081/lxml-6.1.2-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:4b0fa7109b1d0bc1747d8241a0853e135eefb1c978685241b544c46937383efd", size = 4755176, upload-time = "2026-08-19T05:02:18.705Z" },
-    { url = "https://files.pythonhosted.org/packages/3e/1b/cfda9307388d496e7eeb7493d9455896b8137ed95f51f3d6ae6ddcc14a47/lxml-6.1.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:604f4778632588d7c000e7e19430639dc12fca58b5b6e99edffba7631725ef0e", size = 4979444, upload-time = "2026-08-19T05:02:21.262Z" },
-    { url = "https://files.pythonhosted.org/packages/e5/71/f732c8919c45b7f29acf443288c6e90036877a67bfeeb1acceb0fffa011b/lxml-6.1.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5a096d6a5f96b776a5b020cb45c17c545effd2a3b6639e6fa97bc95537600923", size = 5115887, upload-time = "2026-08-19T05:02:23.62Z" },
-    { url = "https://files.pythonhosted.org/packages/30/00/121d52b944f41e33ea86c62875f902d24982842dc7231ab154ac5a6c6593/lxml-6.1.2-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6454d184d556eaf4cb3d6f69e405d21602d6fdcf08b8d57796824275986c6595", size = 5032418, upload-time = "2026-08-19T05:02:26.114Z" },
-    { url = "https://files.pythonhosted.org/packages/70/19/cadb73c7fe48c7563dc8ab62ea53d5b920c8911bfb808507a6daa82e78d2/lxml-6.1.2-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:8b68f2548259bb04e0b3d5df0c397abe8b0080f5e1ffe4019fb7a8bf01a9339e", size = 5603304, upload-time = "2026-08-19T05:02:28.694Z" },
-    { url = "https://files.pythonhosted.org/packages/13/32/9de126a14d5a5db8c371c5ec869178417db226707b62a47273a95ae6df7f/lxml-6.1.2-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6c9cc4b6532abe154dbdebb42aaba8d52c852919591e45067f5b7d46a0405e88", size = 5228938, upload-time = "2026-08-19T05:02:30.99Z" },
-    { url = "https://files.pythonhosted.org/packages/ff/9b/22dd9e843629ed04652591fb220eb2bf2394d97be3be377d60d8083405d7/lxml-6.1.2-cp314-cp314t-manylinux_2_28_i686.whl", hash = "sha256:57188e441ab24f906bd5a5c14eb55363ab51aa6c0de549f3dd320043721cc118", size = 5317790, upload-time = "2026-08-19T05:02:33.301Z" },
-    { url = "https://files.pythonhosted.org/packages/2c/2c/b12a1dc121f81c280635c721c7bcaa341441fcbe37397f60b8915048aece/lxml-6.1.2-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:d0bfd719c254bbe60ea022cff0e6ffb799a6fa7d4d72852cebe0257957b32d68", size = 4646468, upload-time = "2026-08-19T05:02:35.504Z" },
-    { url = "https://files.pythonhosted.org/packages/57/41/fd87a41edc531e7969c25ab1d6b52b5b041eb108b88f6394d6afb4374396/lxml-6.1.2-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:be6f87cd224254a8f81324e34cc655508b83f1d70458a1a39857ad2aa9925852", size = 5240607, upload-time = "2026-08-19T05:02:37.805Z" },
-    { url = "https://files.pythonhosted.org/packages/6e/30/713ba813b6e6673c6dc34733746516017efcd17949b767b154cc50bccf20/lxml-6.1.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:074a88f70a7360a4a0c5be5d898062cd26f898c25b459efb1bdd43ae700c5a1a", size = 5086495, upload-time = "2026-08-19T05:02:40.099Z" },
-    { url = "https://files.pythonhosted.org/packages/33/f8/6532ce0fecd9c326d06b08274ee075cc28dbc9f5e9285355db8504689114/lxml-6.1.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:9031f5f01452681abf39fdd65f84a70cb01a7572a1bbf570042e826b1232d07b", size = 4758801, upload-time = "2026-08-19T05:02:45.434Z" },
-    { url = "https://files.pythonhosted.org/packages/74/b6/5a1f7833ebaa0dd33c28f6f9755ec6ff3891bf63f097634b44e6da1bb65e/lxml-6.1.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:cfeac14425fc7a6fca7864b774d4ee63547926158f4a18c67d77b2c9a948acf1", size = 5626977, upload-time = "2026-08-19T05:02:48.092Z" },
-    { url = "https://files.pythonhosted.org/packages/e6/20/6ae0fc1b45e20877cdcfb1168ceeaf9abb0fba5ed36bd639a260e7b2101e/lxml-6.1.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:8ec111ff8067325f85c08aa9c2b26179ec0537bb89c003fde31127139f85f82d", size = 5235036, upload-time = "2026-08-19T05:02:50.726Z" },
-    { url = "https://files.pythonhosted.org/packages/47/b4/2bc7b37fbb990ccfb7d30393660741592177224a94e07d842c8da70638e8/lxml-6.1.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:48e912f37c99a297175ba955f55a47c0e1c834b506ef162e52a6e4fe276e6e45", size = 5252270, upload-time = "2026-08-19T05:02:53.454Z" },
-    { url = "https://files.pythonhosted.org/packages/4d/0b/07fb8e1dee29a78e2c5fa5c6c914218be76a6406baff27907429566e90ec/lxml-6.1.2-cp314-cp314t-win32.whl", hash = "sha256:7c444c3a6e8e75334879980eed96568f0e12064c8b1913424eac1805e976736b", size = 3902666, upload-time = "2026-08-19T05:02:55.607Z" },
-    { url = "https://files.pythonhosted.org/packages/58/ab/3371527bd9820aae6f511697c93032ed197b0d8dab0f17818f18d3099637/lxml-6.1.2-cp314-cp314t-win_amd64.whl", hash = "sha256:7f35ba7667004ecdafebbe08da7c9fa06ee6195275bb7ef7a29ee1901e69519c", size = 4401011, upload-time = "2026-08-19T05:02:57.899Z" },
-    { url = "https://files.pythonhosted.org/packages/e6/bb/e6de9b2546a4e6df4fb52fb18921906a8b7a041aba06570995759a4d6d8b/lxml-6.1.2-cp314-cp314t-win_arm64.whl", hash = "sha256:d117f39b28ab8a330a74abdbe61c2255b51973b238db25fd6c2448de1eb2a02d", size = 3823384, upload-time = "2026-08-19T05:03:00.371Z" },
-    { url = "https://files.pythonhosted.org/packages/0e/83/7ff98683e14a148191278728d11ba782c3d5137886d49fd95ab4036efa1b/lxml-6.1.2-cp315-cp315-macosx_10_15_universal2.whl", hash = "sha256:1e3c67b817867c484794d7fe0d73045d7d0c67460c78a0a1249a9e92266e6a0e", size = 8609183, upload-time = "2026-08-19T04:58:32.19Z" },
-    { url = "https://files.pythonhosted.org/packages/24/39/c39f05e8240e98009dd3d4ceb248319d0f36467babc5f90a909ed0c5b68a/lxml-6.1.2-cp315-cp315-macosx_10_15_x86_64.whl", hash = "sha256:d3e97ac4353cca3fbbfa829bc0c6a913771573d1c6d46932d4335c46f2b7796a", size = 4639898, upload-time = "2026-08-19T04:58:39.017Z" },
-    { url = "https://files.pythonhosted.org/packages/c5/bf/25e26b089510940a0777ab334357874569255e50930224c8159cd649e754/lxml-6.1.2-cp315-cp315-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:827438bf6c8292d22a409bb7990d7cffce410f33e7664e46ca74d2ecc26975ef", size = 5037527, upload-time = "2026-08-19T04:58:46.224Z" },
-    { url = "https://files.pythonhosted.org/packages/65/6d/aed3a58a3d662f7367a537fabe8c549f1446dbd043719e0ae8cd53f47819/lxml-6.1.2-cp315-cp315-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c470d192e27f97842a068cf12a1c1296b20ca716c56a9249715c6654bc192d19", size = 5661918, upload-time = "2026-08-19T04:59:02.534Z" },
-    { url = "https://files.pythonhosted.org/packages/a5/ca/706d32b6957c0c2e005a9833e8fc528449196b38d5cfcf9e0fd86a96fb00/lxml-6.1.2-cp315-cp315-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ef0b8ba6e13597f681b2b4924ca9c4e8c88420bf0e21d9a9006c757f2fc39d1f", size = 5249359, upload-time = "2026-08-19T05:04:01.956Z" },
-    { url = "https://files.pythonhosted.org/packages/ac/e9/445ff43f56fcffa06f6f3a7189920c216f3eacef68ef834d4111cdbd86ba/lxml-6.1.2-cp315-cp315-manylinux_2_31_armv7l.whl", hash = "sha256:65c32ddc5d0750129c7b119fb57d48192b76d334c21e6b690d19dfb06b34af79", size = 4704548, upload-time = "2026-08-19T05:04:04.57Z" },
-    { url = "https://files.pythonhosted.org/packages/69/78/20b8b7e79a1b1d9cd4465c332d62962858562b446692f16a27068fa54b85/lxml-6.1.2-cp315-cp315-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:0aa07065497f191ad26c4b587ce5dbb5a7105285a3789aafd0661750e8bac537", size = 5261170, upload-time = "2026-08-19T05:04:07.336Z" },
-    { url = "https://files.pythonhosted.org/packages/54/ca/84a0e1148bf511e12e0d99732a4e136a3bf1b91622f0a1b197796e2ff984/lxml-6.1.2-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:cde6b8db7d2e5135129eb5e74b7b44dd2053aa767cd5023541fccedddc262453", size = 5090576, upload-time = "2026-08-19T05:04:09.968Z" },
-    { url = "https://files.pythonhosted.org/packages/6f/f8/1ef6fc7070bed8753315f2e4ea66bc0d37620e1444d014db7f0267b8faaf/lxml-6.1.2-cp315-cp315-musllinux_1_2_armv7l.whl", hash = "sha256:b28842b30c4bc2e6afe137d98a5d2071a62589471e76d053bea55b0e53298af9", size = 4744614, upload-time = "2026-08-19T05:04:12.717Z" },
-    { url = "https://files.pythonhosted.org/packages/87/f6/3a4824cd1c1b81d996d2d75bbd176ba13fbe9b5d89489290d93ff9558486/lxml-6.1.2-cp315-cp315-musllinux_1_2_ppc64le.whl", hash = "sha256:11f529062255209a421ae4de5b1bb36b2f0a2e1a700745e675a4bf4084d13c00", size = 5685792, upload-time = "2026-08-19T05:04:15.367Z" },
-    { url = "https://files.pythonhosted.org/packages/64/9a/f133bf16a67149e00ca5d8a8f1ae662c30a86c303aa242693b67f8e19856/lxml-6.1.2-cp315-cp315-musllinux_1_2_riscv64.whl", hash = "sha256:f8b89b3be75a37509602b03f9cfa1a28298d4eed4625748148307aeb907901b7", size = 5248972, upload-time = "2026-08-19T05:04:18.491Z" },
-    { url = "https://files.pythonhosted.org/packages/50/63/273e7e8a73a5d183d8552dfdaa131dfda0292ddab7bcddc5a66a0ae525d8/lxml-6.1.2-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:1a2331da06dd55a8184985306eb2afd72d708283ce7e85d67bba77317b785060", size = 5271809, upload-time = "2026-08-19T05:04:21.448Z" },
-    { url = "https://files.pythonhosted.org/packages/49/eb/614117c36a28909e79ff7cdec87008f0bd996478f35cf72309189cf398b1/lxml-6.1.2-cp315-cp315-win32.whl", hash = "sha256:442766b326d9892585a64e8c6c4b5ab81d0e6c0538c9f0fc11a84dc101a5d97f", size = 3662854, upload-time = "2026-08-19T05:05:07.141Z" },
-    { url = "https://files.pythonhosted.org/packages/5c/e9/06aee6107cf8e7b870f10f82539f366cba10dc6053144cca80e838caf8c8/lxml-6.1.2-cp315-cp315-win_amd64.whl", hash = "sha256:a7fd1dd6faa3df9dcd8f1765237362cd885ca62cdf77a7c5f5ea383ae5b6048b", size = 4074590, upload-time = "2026-08-19T05:05:09.697Z" },
-    { url = "https://files.pythonhosted.org/packages/84/bf/dad9b6baf9b26d79584834e15cef2a5dd0a13c7b1df08831e8f18244b494/lxml-6.1.2-cp315-cp315-win_arm64.whl", hash = "sha256:054175250531a5fb102d485743ff16412279c93add12385b3b1c3d7b16d8deaa", size = 3749336, upload-time = "2026-08-19T05:05:12.334Z" },
-    { url = "https://files.pythonhosted.org/packages/5f/9d/cd0c43d45e2eb52df7735c6558f24054ca633499191899b0cb9040fbbc3c/lxml-6.1.2-cp315-cp315t-macosx_10_15_universal2.whl", hash = "sha256:84a2a46b93b789d8acb44cfcb3d967ce9dbe29884ddb93fbb1a33f0e0c8fcd86", size = 8857688, upload-time = "2026-08-19T05:04:24.648Z" },
-    { url = "https://files.pythonhosted.org/packages/0b/26/27093dc1a9edbdd8a54652f237a387f7e63ec0192efe708bc2576d8a1383/lxml-6.1.2-cp315-cp315t-macosx_10_15_x86_64.whl", hash = "sha256:4aced3284e0353c798b060fe2c175eb81410e99b9a7e2ae6951be5333732b111", size = 4754422, upload-time = "2026-08-19T05:04:27.645Z" },
-    { url = "https://files.pythonhosted.org/packages/2c/ee/502f7c93507f57eb496744a64da8f4ca86855cf88e48d14584342f1bfd92/lxml-6.1.2-cp315-cp315t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:47c92dc5167de16e27ace8332454f12ba172dcab04f7a78a9eae14e2e41b6a41", size = 5033396, upload-time = "2026-08-19T05:04:30.054Z" },
-    { url = "https://files.pythonhosted.org/packages/bf/72/c4cbbe72f951650f2afe43a70e51687e111d82b9bec46e3310ea76419d46/lxml-6.1.2-cp315-cp315t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:40366c23a938008a3bedfcfd80709b3a857c188b4d710b083e978ef5d2c1c715", size = 5615298, upload-time = "2026-08-19T05:04:32.752Z" },
-    { url = "https://files.pythonhosted.org/packages/14/83/a3df966d6d7b6513e9dfb6fbfb041c0619642170359c1b36ab20a83e59eb/lxml-6.1.2-cp315-cp315t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1c4c6dc1b2485aaa4adfb6ed754f90dddcb2b96a66bbebc9e1ac242b5ce5e818", size = 5236282, upload-time = "2026-08-19T05:04:35.762Z" },
-    { url = "https://files.pythonhosted.org/packages/4f/85/8692ec8173c9f8d295735b9bf410d202317e7b3ed11141e80a30f421f409/lxml-6.1.2-cp315-cp315t-manylinux_2_31_armv7l.whl", hash = "sha256:3a698fad6f122a9b3e2dc2fb598c1de7329c74a67c7a334c9109a440de2508e5", size = 4650647, upload-time = "2026-08-19T05:04:38.396Z" },
-    { url = "https://files.pythonhosted.org/packages/11/e7/dbe3cece28a5bf82997a091d9dbb0fc49e725a5fa86550897ee2cf6412e6/lxml-6.1.2-cp315-cp315t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:14879fa5eb2b793c040bbfcb62011aa3015c65d6c9875e063ea98ce2029d51fb", size = 5243387, upload-time = "2026-08-19T05:04:41.247Z" },
-    { url = "https://files.pythonhosted.org/packages/99/a9/81a2d27640db0d27200b2f32339a54e74c36d58feb5ad528b87d52a59ecc/lxml-6.1.2-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:b631174cd2e4d9f8a94ef17f911c6ded10ede93b5e7860dee7bbf85961d321e9", size = 5092624, upload-time = "2026-08-19T05:04:43.919Z" },
-    { url = "https://files.pythonhosted.org/packages/cb/f4/0b0304c70c087f618d95b0306738b070bd556afd09c2c92589b78dbe5eb0/lxml-6.1.2-cp315-cp315t-musllinux_1_2_armv7l.whl", hash = "sha256:ceafa5e0536c62a5cd9f65327fa0b57d6f0b0e3435daf2c98a78d0dde7ecbae1", size = 4758742, upload-time = "2026-08-19T05:04:46.615Z" },
-    { url = "https://files.pythonhosted.org/packages/a2/ec/f9fc45f1d01b632b673e11880e75292dff9953db9f426d1a38201b8eb5f5/lxml-6.1.2-cp315-cp315t-musllinux_1_2_ppc64le.whl", hash = "sha256:7c482e87cc86bed78a50462560675bc2c348ef72c47596f9b933346d5a8e920e", size = 5649540, upload-time = "2026-08-19T05:04:49.777Z" },
-    { url = "https://files.pythonhosted.org/packages/6a/0b/d65e0458c2bcce0df68d5cc29ad0006e76446f02d9e50caf188fd1fb8bae/lxml-6.1.2-cp315-cp315t-musllinux_1_2_riscv64.whl", hash = "sha256:b1c0d2dde8a50520efc51644587f0fc4810e3af7d3e029d7af0be93bf39e2b5c", size = 5234869, upload-time = "2026-08-19T05:04:52.972Z" },
-    { url = "https://files.pythonhosted.org/packages/ee/62/1fee828238badd3bfe9544f5cc9ce6ded421ef38e9634030445dedd78b36/lxml-6.1.2-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:dd7ea3fa47154b9fff90591b961e41b3718bd7fcd5bc2d9bb47e9845c8ace088", size = 5259992, upload-time = "2026-08-19T05:04:56.028Z" },
-    { url = "https://files.pythonhosted.org/packages/20/18/35fb14dd6baccbffa6daeb2369802f04a94e3f73db3c7bb405dbab009729/lxml-6.1.2-cp315-cp315t-win32.whl", hash = "sha256:87534cec6ea325435e4adf2326b0cf3110eee9a47abf73652eb155db639c08c6", size = 3901151, upload-time = "2026-08-19T05:04:58.671Z" },
-    { url = "https://files.pythonhosted.org/packages/f0/b6/07530896ca062bc3d2f09d5cb8a48e799c05b12c496205db03159ba13b6c/lxml-6.1.2-cp315-cp315t-win_amd64.whl", hash = "sha256:4e220a9c297e5d36895d489a08c9a3f1f6193b6414e702c5fb751e4a3767f8d0", size = 4395355, upload-time = "2026-08-19T05:05:01.651Z" },
-    { url = "https://files.pythonhosted.org/packages/31/a8/237d8de1d77085cfd41d0c6049a044d8d01886f3afb7f1eda2f43d900a96/lxml-6.1.2-cp315-cp315t-win_arm64.whl", hash = "sha256:f16a407766bac51c65d605b06d900821751a79aa20e12185f273f14a17180e7b", size = 3822823, upload-time = "2026-08-19T05:05:04.63Z" },
-]
-
 [[package]]
 name = "markdown-it-py"
 version = "4.2.0"
@@ -1720,15 +1520,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/0f/54/68a0978d1ef8502b8492099beaa6e7a0c1b32e3b5d4f677f5810cb08711c/pandas-3.0.3-cp314-cp314t-win_arm64.whl", hash = "sha256:b2c95f8bfc1ee412bf482605d7bfd30c12d1d26bd59fdd91efeef1d4718decb1", size = 9466464, upload-time = "2026-05-11T18:54:22.754Z" },
 ]
 
-[[package]]
-name = "parse"
-version = "1.22.1"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/a4/f2/0b504486c2a5564798607d3860e48ed19c6443d5e9cc3ec61cc6b8b4ef58/parse-1.22.1.tar.gz", hash = "sha256:d3a4740ec3da338e2b258b2d69741b731eadfddca59e24a14bc4ee5fce38c911", size = 36970, upload-time = "2026-05-26T03:44:52.624Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/6f/c5/7c16e99869e1f422629092cfd23e3b58e461988c3f9c36fd3624bb4142e6/parse-1.22.1-py2.py3-none-any.whl", hash = "sha256:20f0925a46f06602485ac90d751764d0697fd8455aaa97489ba8953a4b66de32", size = 20925, upload-time = "2026-05-26T03:44:51.156Z" },
-]
-
 [[package]]
 name = "parso"
 version = "0.8.7"
@@ -2167,20 +1958,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/9d/7a/d968e294073affff457b041c2be9868a40c1c71f4a35fcc1e45e5493067b/pytest_cov-7.1.0-py3-none-any.whl", hash = "sha256:a0461110b7865f9a271aa1b51e516c9a95de9d696734a2f71e3e78f46e1d4678", size = 22876, upload-time = "2026-03-21T20:11:14.438Z" },
 ]
 
-[[package]]
-name = "python-daemon"
-version = "2.1.2"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "docutils" },
-    { name = "lockfile" },
-    { name = "setuptools" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/b2/fb/a280d65f81e9d69989c8d6c4e0bb18d7280cdcd6d406a2cc3f4eb47d4402/python-daemon-2.1.2.tar.gz", hash = "sha256:261c859be5c12ae7d4286dc6951e87e9e1a70a882a8b41fd926efc1ec4214f73", size = 76176, upload-time = "2016-10-26T10:16:01.909Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/f4/59/816004688f8e8602526553cd96226f34657ce4a86daa2240c3eebb0568a3/python_daemon-2.1.2-py2.py3-none-any.whl", hash = "sha256:53da55aec3bb67b576e13a8091a2181f99b395c2eec32a5a0d91d347a5c420a7", size = 19425, upload-time = "2016-10-26T10:16:05.441Z" },
-]
-
 [[package]]
 name = "python-dateutil"
 version = "2.9.0.post0"
@@ -2473,15 +2250,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/89/d3/726bd88f0eece09ddf431bea4c9191c18e7a8d070b854eb0014d447712ee/sentry_sdk-2.66.1-py3-none-any.whl", hash = "sha256:86002793161d9a95ef04bdd8d442e9bfece5d989b755f05d6360215094a7aff6", size = 505555, upload-time = "2026-07-22T12:26:52.71Z" },
 ]
 
-[[package]]
-name = "setuptools"
-version = "84.0.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/6d/44/f5da03a8ef95d369145c5bb53050e7877c9f3d312e128605fd9504829143/setuptools-84.0.0.tar.gz", hash = "sha256:f4695c21257f0d9b537ec2692c941d02ee143b7cc1276941349a546573b2ef73", size = 1168449, upload-time = "2026-08-08T18:27:58.365Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/95/9c/c510029fc6ef33a6275cd2c5d3cecd6613dfd6aa401d57c54f1c18852ccf/setuptools-84.0.0-py3-none-any.whl", hash = "sha256:51a52592b3b99e102b609654876bd65f19f999935166d1352678931132b0c670", size = 818216, upload-time = "2026-08-08T18:27:56.719Z" },
-]
-
 [[package]]
 name = "shellingham"
 version = "1.5.4"
@@ -2500,15 +2268,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" },
 ]
 
-[[package]]
-name = "smmap"
-version = "5.0.3"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/1f/ea/49c993d6dfdd7338c9b1000a0f36817ed7ec84577ae2e52f890d1a4ff909/smmap-5.0.3.tar.gz", hash = "sha256:4d9debb8b99007ae47165abc08670bd74cb74b5227dda7f643eccc4e9eb5642c", size = 22506, upload-time = "2026-03-09T03:43:26.1Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/c1/d4/59e74daffcb57a07668852eeeb6035af9f32cbfd7a1d2511f17d2fe6a738/smmap-5.0.3-py3-none-any.whl", hash = "sha256:c106e05d5a61449cf6ba9a1e650227ecfb141590d2a98412103ff35d89fc7b2f", size = 24390, upload-time = "2026-03-09T03:43:24.361Z" },
-]
-
 [[package]]
 name = "stack-data"
 version = "0.6.3"
@@ -2546,15 +2305,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/9b/24/84ce997e8ae6296168a74d0d9c4dde572d90fb23fd7c0b219c30ff71e00e/tbb-2021.13.1-py3-none-win_amd64.whl", hash = "sha256:cbf024b2463fdab3ebe3fa6ff453026358e6b903839c80d647e08ad6d0796ee9", size = 286908, upload-time = "2024-08-07T15:09:05.677Z" },
 ]
 
-[[package]]
-name = "tenacity"
-version = "8.5.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/a3/4d/6a19536c50b849338fcbe9290d562b52cbdcf30d8963d3588a68a4107df1/tenacity-8.5.0.tar.gz", hash = "sha256:8bc6c0c8a09b31e6cad13c47afbed1a567518250a9a171418582ed8d9c20ca78", size = 47309, upload-time = "2024-07-05T07:25:31.836Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/d2/3f/8ba87d9e287b9d385a02a7114ddcef61b26f86411e121c9003eb509a1773/tenacity-8.5.0-py3-none-any.whl", hash = "sha256:b594c2a5945830c267ce6b79a166228323ed52718f30302c1359836112346687", size = 28165, upload-time = "2024-07-05T07:25:29.591Z" },
-]
-
 [[package]]
 name = "threadpoolctl"
 version = "3.6.0"
@@ -2848,20 +2598,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/bd/6e/95b0e537de1f4d4301f76f944642c6da50d1511cc7b3d64dc418a66c7509/wcwidth-0.8.1-py3-none-any.whl", hash = "sha256:f453740b1e4a4f3291faa37944c555d71056c4da08d59809b307ef4feba695c8", size = 323092, upload-time = "2026-06-08T05:57:21.413Z" },
 ]
 
-[[package]]
-name = "webdavclient3"
-version = "3.14.7"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "lxml" },
-    { name = "python-dateutil" },
-    { name = "requests" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/67/d8/ca3981053ed553363322f71745f543186b93439b6417f5d6ca91d4b4fec7/webdavclient3-3.14.7.tar.gz", hash = "sha256:6c04252b579bc015cec78081480c63eadf1030f382768248777c6203f059b3f5", size = 30836, upload-time = "2026-02-06T17:54:15.506Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/89/5e/0b1c2f494d03c4acbc44567fa68b954cd0fa3f21eb3f9528011da371f9b1/webdavclient3-3.14.7-py3-none-any.whl", hash = "sha256:a904381da8e3ae77b4ca9e11e05058d91a07704254d71c193c797f7c2fb15025", size = 22887, upload-time = "2026-02-06T17:54:14.068Z" },
-]
-
 [[package]]
 name = "xxhash"
 version = "3.7.0"