lars/giant
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Unify dataset/tooling scripts into a single dwarf Typer CLI

Browse Source 
Replace the five separately-hyphenated uv entry points (steps-to-parquet,
steps-to-parquet-parallel, migrate-geant-steps, bump-dataset-version,
create-root-files) plus the unregistered hparam_scan.py with one `dwarf`
command exposing convert/migrate/bump-gen/bump-schema/status/
update-manifest/create-manifest/make-root/hparam-scan as subcommands.

Each scripts/*.py module now only holds argparse-free business logic;
scripts/dwarf.py wires it up with Typer, matching giant/cli.py's style.
`dwarf convert` merges the old serial/parallel conversion scripts behind
a --jobs flag (default 1: sequential with plain -o; >1: dataset-layout
fan-out via subprocess).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
lars
2026-07-02 09:52:37 +02:00
co-authored by Claude Sonnet 5
parent 7b37b284f8
commit d5853d5a75
13 changed files with 662 additions and 505 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+11 -8
View File
@@ -34,7 +34,7 @@ The model is developed in two phases:
## Data
Input: parquet files produced by [miniCaloSim](https://gitlab.etp.kit.edu/lbogner/minicalosim), or converted from a ROOT file via `uv run steps-to-parquet`. Each row is one Geant4 step. Train/val split is by `event_id` (not row shuffle) to avoid leaking correlated steps from the same shower.
Input: parquet files produced by [miniCaloSim](https://gitlab.etp.kit.edu/lbogner/minicalosim), or converted from a ROOT file via `uv run dwarf convert`. Each row is one Geant4 step. Train/val split is by `event_id` (not row shuffle) to avoid leaking correlated steps from the same shower.
## Project structure
@@ -55,13 +55,16 @@ giant/
│ ├── validate.py # step-level marginal + KL-divergence validation
│ ├── analysis.py # notebook diagnostics: marginals, correlations, constraint checks
│ └── cli.py # `giant train` / `giant predict` Typer app
├── scripts/ # also exposed as uv entry points, e.g. `uv run steps-to-parquet`
│ ├── steps_to_parquet.py # ROOT → parquet conversion (uproot/awkward/polars)
│ ├── steps_to_parquet_parallel.py # fan out steps_to_parquet.py over several ROOT files
│ ├── migrate_geant_steps.py # one-time move into the raw/processed/pools/derived layout
│ ├── bump_dataset_version.py # cut a new raw gen or parquet schema, with a logged reason
│ ├── create_root_files.py # generate new ROOT shards via a minicalosim executable
│ └── hparam_scan.py # hyperparameter grid scan over `giant train` runs
├── scripts/ # dataset/tooling logic, unified under the `dwarf` CLI (`uv run dwarf --help`)
│ ├── dwarf.py # Typer app: convert, migrate, bump-gen, bump-schema, status,
│ │ # update-manifest, create-manifest, make-root, hparam-scan
│ ├── steps_to_parquet.py # ROOT → parquet conversion (uproot/awkward/polars) — `dwarf convert`
│ ├── steps_to_parquet_parallel.py # fan out conversion over several ROOT files — `dwarf convert --jobs N`
│ ├── migrate_geant_steps.py # one-time move into the raw/processed/pools/derived layout — `dwarf migrate`
│ ├── bump_dataset_version.py # cut a new raw gen or parquet schema, with a logged reason —
│ │ # `dwarf bump-gen` / `bump-schema` / `status` / `update-manifest` / `create-manifest`
│ ├── create_root_files.py # generate new ROOT shards via a minicalosim executable — `dwarf make-root`
│ └── hparam_scan.py # hyperparameter grid scan over `giant train` runs — `dwarf hparam-scan`
└── tests/
```