analyze: default run directory to <repo>/analysis_runs, gitignored
CI / Lint (ruff check) (push) Successful in 1m0s
CI / Format (ruff format) (push) Successful in 1m7s
CI / Type check (ty) (push) Successful in 1m11s
CI / Tests (push) Successful in 1m49s
CI / Lint (ruff check) (pull_request) Successful in 1m6s
CI / Format (ruff format) (pull_request) Successful in 58s
CI / Type check (ty) (pull_request) Successful in 1m7s
CI / Tests (pull_request) Successful in 1m50s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped
CI / Lint (ruff check) (push) Successful in 1m0s
CI / Format (ruff format) (push) Successful in 1m7s
CI / Type check (ty) (push) Successful in 1m11s
CI / Tests (push) Successful in 1m49s
CI / Lint (ruff check) (pull_request) Successful in 1m6s
CI / Format (ruff format) (pull_request) Successful in 58s
CI / Type check (ty) (pull_request) Successful in 1m7s
CI / Tests (pull_request) Successful in 1m50s
CI / Bump version, build & publish wheel (push) Has been skipped
CI / Bump version, build & publish wheel (pull_request) Has been skipped
giant analyze prep/submit previously defaulted the run directory to next to the rollout parquet on /ceph. Default it instead to <cwd>/analysis_runs/analysis_<id> so it lands inside the portal repo checkout (/work) — gitignored, --run-dir still overrides it. derive_run_dir/prep gained a default_base param; library callers that don't pass one keep the old parquet-relative fallback.
This commit is contained in:
@@ -93,13 +93,26 @@ def load_rollout_yaml(path: str | Path) -> dict:
|
||||
return d
|
||||
|
||||
|
||||
def derive_run_dir(rollout_yaml: dict, run_dir: str | Path | None = None) -> Path:
|
||||
"""Analysis output directory, next to the rollout parquet unless overridden."""
|
||||
def derive_run_dir(
|
||||
rollout_yaml: dict,
|
||||
run_dir: str | Path | None = None,
|
||||
default_base: str | Path | None = None,
|
||||
) -> Path:
|
||||
"""Analysis output directory.
|
||||
|
||||
Precedence: an explicit ``run_dir`` always wins. Otherwise
|
||||
``default_base / analysis_<tag>`` if ``default_base`` is given (the CLI
|
||||
passes the repo's gitignored ``analysis_runs/``, so run directories don't
|
||||
pile up on ``/ceph`` next to the rollout parquet). Falls back to next to
|
||||
the rollout parquet — the original convention — for callers that don't
|
||||
care where the run directory lives.
|
||||
"""
|
||||
if run_dir is not None:
|
||||
return Path(run_dir)
|
||||
rollout = Path(rollout_yaml["output"])
|
||||
tag = str(rollout_yaml.get("prediction_id") or rollout.stem)[:8]
|
||||
return rollout.parent / f"analysis_{tag}"
|
||||
base = Path(default_base) if default_base is not None else rollout.parent
|
||||
return base / f"analysis_{tag}"
|
||||
|
||||
|
||||
def _plot_meta(rollout_yaml: dict) -> dict:
|
||||
@@ -160,6 +173,7 @@ def prep(
|
||||
rollout_yaml: str | Path,
|
||||
run_dir: str | Path | None = None,
|
||||
n_chunks: int = 1,
|
||||
default_base: str | Path | None = None,
|
||||
**ctx_kwargs,
|
||||
) -> Path:
|
||||
"""Read the rollout YAML, build the shared context, and lay out the run dir.
|
||||
@@ -168,10 +182,11 @@ def prep(
|
||||
``n_chunks`` is the run-level chunk count every ``compute-one``/``merge-one``
|
||||
job reads back out of ``run_meta.json`` (via ``RunMeta.n_chunks``), so it is
|
||||
resolved once here rather than re-passed (and risking disagreement) at every
|
||||
later step.
|
||||
later step. See ``derive_run_dir`` for how ``run_dir``/``default_base``
|
||||
resolve the actual directory.
|
||||
"""
|
||||
y = load_rollout_yaml(rollout_yaml)
|
||||
run_path = derive_run_dir(y, run_dir)
|
||||
run_path = derive_run_dir(y, run_dir, default_base=default_base)
|
||||
run_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
rollout, reference = y["output"], y["dataset"]
|
||||
|
||||
Reference in New Issue
Block a user