- run_create_manifest gains --force; it previously overwrote an
existing manifest (including holdout.manifest, which
check_holdout_overlap exists specifically to protect) with no
warning or backup on a second run.
- _git_user_name only caught OSError, not subprocess.TimeoutExpired (a
SubprocessError, not an OSError) — a slow/loaded shared portal
machine could crash `dwarf bump-gen`/`bump-schema` instead of
degrading to by=None as intended.
- `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when
the requested count exceeds ~1/4 of the machine's CPUs, matching the
same shared-machine etiquette check added to giant train in the
previous commit.
- The Conditioning enum was independently redefined in both
giant/cli.py and scripts/dwarf.py; moved to a single
giant.config.Conditioning both now import, removing the drift risk
of a third conditioning mode being added to one but not the other.
Each fix has a regression test.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Removes unused imports and an ambiguous variable name, narrows
Optional types before use so ty's flow analysis is satisfied, swaps
sum() over polars expressions for pl.sum_horizontal to avoid the
Literal[0] fallback type, and converts numpy bin edges to plain lists
before passing to matplotlib's hist (whose stub only accepts
Sequence[float]). Also applies ruff format across the repo, which had
drifted out of sync with the formatter.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
origin/energy-conservation-poc grew bump-gen/bump-schema --to and
update-manifest --gen flags (091b23a) plus a train output-dir date
prefix (305e436) after the dwarf unification was written locally.
Reconcile: bring plan_bump_gen/plan_bump_schema/plan_update_manifest's
target/target_gen support into the plain-function (argparse-free) form,
thread --to/--gen through scripts/dwarf.py's bump-gen/bump-schema/
update-manifest commands, and take giant/cli.py's date-prefix change
and the associated tests as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
dev and full are allowed to share files — only holdout must be strictly
isolated. When creating dev or full, only compare against holdout.manifest;
when creating holdout, compare against all other manifests in the dir.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
update-manifest rewrites the schemaN component in existing manifest files to a
specified or auto-detected highest schema, verifying all target files exist before
writing. create-manifest builds a new manifest from explicit parquet file paths,
supporting --pool/--type (full|holdout|dev) to derive the output path from root,
and enforcing holdout isolation by checking for cross-manifest overlap whenever a
holdout manifest is involved.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
scripts/ is now a proper package (scripts/__init__.py, added to the wheel's
packages), with each script registered under [project.scripts] using its
bare dashed name (e.g. `uv run migrate-geant-steps`). Tests now import these
modules normally instead of loading them by file path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduces raw/<kind>/<gen>/<detector>/shard-NNN.root and
processed/<kind>/<gen>/<schema>/<detector>/shard-NNN.parquet as the dataset
convention, plus scripts to operate on it: migrate_geant_steps.py for the
one-time move into this layout, bump_dataset_version.py to cut new
gen/schema versions with a logged reason, steps_to_parquet_parallel.py to
convert ROOT shards to parquet in parallel and place them correctly, and
create_root_files.py to generate new ROOT shards via a minicalosim
executable. The loader gains .manifest file support so pools/ (train/dev/
holdout shard lists) can be passed straight to `giant train`.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lars