analyze: chunk per-plot aggregation across HTCondor jobs
CI / Lint (ruff check) (push) Successful in 1m1s
CI / Format (ruff format) (push) Successful in 1m5s
CI / Type check (ty) (push) Successful in 1m6s
CI / Tests (push) Successful in 1m52s
CI / Lint (ruff check) (pull_request) Successful in 1m3s
CI / Format (ruff format) (pull_request) Successful in 1m4s
CI / Type check (ty) (pull_request) Successful in 1m4s
CI / Tests (pull_request) Successful in 1m42s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped
CI / Lint (ruff check) (push) Successful in 1m1s
CI / Format (ruff format) (push) Successful in 1m5s
CI / Type check (ty) (push) Successful in 1m6s
CI / Tests (push) Successful in 1m52s
CI / Lint (ruff check) (pull_request) Successful in 1m3s
CI / Format (ruff format) (pull_request) Successful in 1m4s
CI / Type check (ty) (pull_request) Successful in 1m4s
CI / Tests (pull_request) Successful in 1m42s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped
Add a second parallelism axis to giant analyze: each plot's data can now be split into a configurable number of event_id-disjoint chunks, each computed as its own HTCondor job, bounding per-job walltime and scan cost on large rollout/reference files instead of one job re-scanning the whole file per plot. Every PlotSpec now splits into compute_partial (runs per (plot, chunk) job against a chunk-filtered Bundle) and finalize (merges chunks - elementwise sum for fixed-edge histograms/species shares, concatenate -then-recompute for specs that derive edges or mean/std from the full per-event/per-secondary array). Router diagnostics stay chunkable=False and always run as a single job. giant analyze render now joins every plot's chunk partials (merge_all) before rendering, transparently. New: --chunks on `analyze prep`/`analyze submit`, --chunk on `analyze compute-one`, and a new `analyze merge-one` command.
This commit is contained in:
@@ -303,12 +303,16 @@ def render_all(
|
||||
def render_run(run_dir: str | Path, *, run_gallery: bool = False) -> list[Path]:
|
||||
"""Render a prepped run directory: ``<run_dir>/reduced`` → ``<run_dir>/plots``.
|
||||
|
||||
Pulls the rollout provenance (checkpoint, paths, cutoffs) from
|
||||
``run_meta.json`` into every plot's gallery metadata.
|
||||
First joins every plot's chunk partials (``reduced_partial/<id>__*.json``)
|
||||
into ``reduced/<id>.json`` via ``merge_all`` — a no-op merge when the run
|
||||
wasn't chunked (``n_chunks=1``) — then pulls the rollout provenance
|
||||
(checkpoint, paths, cutoffs) from ``run_meta.json`` into every plot's
|
||||
gallery metadata and renders.
|
||||
"""
|
||||
from giant.analysis.condor import RunMeta
|
||||
from giant.analysis.condor import RunMeta, merge_all
|
||||
|
||||
run_dir = Path(run_dir)
|
||||
merge_all(run_dir)
|
||||
meta = RunMeta.load(run_dir / "run_meta.json")
|
||||
run_meta = {
|
||||
"title": meta.title,
|
||||
|
||||
Reference in New Issue
Block a user