analyze: chunk per-plot aggregation across HTCondor jobs
CI / Lint (ruff check) (push) Successful in 1m1s
CI / Format (ruff format) (push) Successful in 1m5s
CI / Type check (ty) (push) Successful in 1m6s
CI / Tests (push) Successful in 1m52s
CI / Lint (ruff check) (pull_request) Successful in 1m3s
CI / Format (ruff format) (pull_request) Successful in 1m4s
CI / Type check (ty) (pull_request) Successful in 1m4s
CI / Tests (pull_request) Successful in 1m42s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped

Add a second parallelism axis to giant analyze: each plot's data can now
be split into a configurable number of event_id-disjoint chunks, each
computed as its own HTCondor job, bounding per-job walltime and scan cost
on large rollout/reference files instead of one job re-scanning the
whole file per plot.

Every PlotSpec now splits into compute_partial (runs per (plot, chunk)
job against a chunk-filtered Bundle) and finalize (merges chunks -
elementwise sum for fixed-edge histograms/species shares, concatenate
-then-recompute for specs that derive edges or mean/std from the full
per-event/per-secondary array). Router diagnostics stay chunkable=False
and always run as a single job. giant analyze render now joins every
plot's chunk partials (merge_all) before rendering, transparently.

New: --chunks on `analyze prep`/`analyze submit`, --chunk on
`analyze compute-one`, and a new `analyze merge-one` command.
This commit is contained in:
2026-07-27 09:24:19 +02:00
parent 70d0f04326
commit 86fc46b5a8
10 changed files with 787 additions and 192 deletions
+7 -3
View File
@@ -303,12 +303,16 @@ def render_all(
def render_run(run_dir: str | Path, *, run_gallery: bool = False) -> list[Path]:
"""Render a prepped run directory: ``<run_dir>/reduced`` → ``<run_dir>/plots``.
Pulls the rollout provenance (checkpoint, paths, cutoffs) from
``run_meta.json`` into every plot's gallery metadata.
First joins every plot's chunk partials (``reduced_partial/<id>__*.json``)
into ``reduced/<id>.json`` via ``merge_all`` — a no-op merge when the run
wasn't chunked (``n_chunks=1``) — then pulls the rollout provenance
(checkpoint, paths, cutoffs) from ``run_meta.json`` into every plot's
gallery metadata and renders.
"""
from giant.analysis.condor import RunMeta
from giant.analysis.condor import RunMeta, merge_all
run_dir = Path(run_dir)
merge_all(run_dir)
meta = RunMeta.load(run_dir / "run_meta.json")
run_meta = {
"title": meta.title,