lars/giant
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
203 Commits 2 Branches 24 Tags
9ce55e501319b5650f461c8b7070be0355835c76
Commit Graph
7 Commits
Author SHA1 Message Date
larsandClaude Sonnet 5 ca3a2a3462 Fix CLI/tooling robustness gaps and dedupe the Conditioning enum
CI / Lint (ruff check) (push) Successful in 35s
Details
CI / Format (ruff format) (push) Failing after 31s
Details
CI / Type check (ty) (push) Successful in 30s
Details
CI / Sync project version with tag (push) Has been skipped
Details
CI / Tests (push) Successful in 1m7s
Details 
- run_create_manifest gains --force; it previously overwrote an
  existing manifest (including holdout.manifest, which
  check_holdout_overlap exists specifically to protect) with no
  warning or backup on a second run.
- _git_user_name only caught OSError, not subprocess.TimeoutExpired (a
  SubprocessError, not an OSError) — a slow/loaded shared portal
  machine could crash `dwarf bump-gen`/`bump-schema` instead of
  degrading to by=None as intended.
- `dwarf convert --jobs`/`make-root --jobs` now warn (never block) when
  the requested count exceeds ~1/4 of the machine's CPUs, matching the
  same shared-machine etiquette check added to giant train in the
  previous commit.
- The Conditioning enum was independently redefined in both
  giant/cli.py and scripts/dwarf.py; moved to a single
  giant.config.Conditioning both now import, removing the drift risk
  of a third conditioning mode being added to one but not the other.

Each fix has a regression test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
 2026-08-03 13:49:09 +02:00
larsandClaude Sonnet 5 25718f175e Fix ruff, ty, and pytest failures; apply ruff format 
Removes unused imports and an ambiguous variable name, narrows
Optional types before use so ty's flow analysis is satisfied, swaps
sum() over polars expressions for pl.sum_horizontal to avoid the
Literal[0] fallback type, and converts numpy bin edges to plain lists
before passing to matplotlib's hist (whose stub only accepts
Sequence[float]). Also applies ruff format across the repo, which had
drifted out of sync with the formatter.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
 2026-07-02 16:59:22 +02:00
larsandClaude Sonnet 5 917835e182 Fold --to/--gen dataset-versioning flags into the dwarf CLI 
origin/energy-conservation-poc grew bump-gen/bump-schema --to and
update-manifest --gen flags (091b23a) plus a train output-dir date
prefix (305e436) after the dwarf unification was written locally.
Reconcile: bring plan_bump_gen/plan_bump_schema/plan_update_manifest's
target/target_gen support into the plain-function (argparse-free) form,
thread --to/--gen through scripts/dwarf.py's bump-gen/bump-schema/
update-manifest commands, and take giant/cli.py's date-prefix change
and the associated tests as-is.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
 2026-07-02 09:57:17 +02:00
larsandClaude Sonnet 4.6 caaf3c6102 Restrict holdout overlap check to holdout vs dev/full only 
dev and full are allowed to share files — only holdout must be strictly
isolated. When creating dev or full, only compare against holdout.manifest;
when creating holdout, compare against all other manifests in the dir.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-26 11:24:05 +02:00
larsandClaude Sonnet 4.6 f56a45690d Add update-manifest and create-manifest subcommands to bump_dataset_version 
update-manifest rewrites the schemaN component in existing manifest files to a
specified or auto-detected highest schema, verifying all target files exist before
writing. create-manifest builds a new manifest from explicit parquet file paths,
supporting --pool/--type (full|holdout|dev) to derive the output path from root,
and enforcing holdout isolation by checking for cross-manifest overlap whenever a
holdout manifest is involved.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-26 11:03:32 +02:00
larsandClaude Sonnet 4.6 5be91e0d17 Expose dataset/conversion scripts as uv entry points 
scripts/ is now a proper package (scripts/__init__.py, added to the wheel's
packages), with each script registered under [project.scripts] using its
bare dashed name (e.g. `uv run migrate-geant-steps`). Tests now import these
modules normally instead of loading them by file path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-25 16:51:47 +02:00
larsandClaude Sonnet 4.6 320365606a Add tooling for a versioned geant_steps dataset layout 
Introduces raw/<kind>/<gen>/<detector>/shard-NNN.root and
processed/<kind>/<gen>/<schema>/<detector>/shard-NNN.parquet as the dataset
convention, plus scripts to operate on it: migrate_geant_steps.py for the
one-time move into this layout, bump_dataset_version.py to cut new
gen/schema versions with a logged reason, steps_to_parquet_parallel.py to
convert ROOT shards to parquet in parallel and place them correctly, and
create_root_files.py to generate new ROOT shards via a minicalosim
executable. The loader gains .manifest file support so pools/ (train/dev/
holdout shard lists) can be passed straight to `giant train`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-25 16:46:48 +02:00