Commit Graph

2 Commits

Author SHA1 Message Date
lars 471a81b5e7 Add dwarf warm-cache to precompute the setup-stage sidecar
CI / Format (ruff format) (push) Successful in 26s
CI / Lint (ruff check) (push) Successful in 27s
CI / Sync project version with tag (push) Has been skipped
CI / Type check (ty) (push) Successful in 37s
CI / Lint (ruff check) (pull_request) Successful in 36s
CI / Format (ruff format) (pull_request) Successful in 31s
CI / Sync project version with tag (pull_request) Has been skipped
CI / Type check (ty) (pull_request) Successful in 33s
CI / Tests (push) Successful in 1m43s
CI / Tests (pull_request) Successful in 1m39s
Lets the vocab maps, event-id split index, and normalizer stats be
warmed once for a dataset (right after `dwarf convert`, or before a
`dwarf hparam-scan` sweep) without needing to also start training.

Extracts the setup-stage logic out of giant/pipeline.py:run_train_job
into a standalone run_setup_stage() (returning a SetupStageResult),
reused by both run_train_job and the new dwarf command's
scripts/warm_setup_cache.py — a behavior-preserving refactor, covered
by the existing test_pipeline.py suite.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-30 10:53:31 +02:00
lars d5853d5a75 Unify dataset/tooling scripts into a single dwarf Typer CLI
Replace the five separately-hyphenated uv entry points (steps-to-parquet,
steps-to-parquet-parallel, migrate-geant-steps, bump-dataset-version,
create-root-files) plus the unregistered hparam_scan.py with one `dwarf`
command exposing convert/migrate/bump-gen/bump-schema/status/
update-manifest/create-manifest/make-root/hparam-scan as subcommands.

Each scripts/*.py module now only holds argparse-free business logic;
scripts/dwarf.py wires it up with Typer, matching giant/cli.py's style.
`dwarf convert` merges the old serial/parallel conversion scripts behind
a --jobs flag (default 1: sequential with plain -o; >1: dataset-layout
fan-out via subprocess).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-02 09:52:37 +02:00