giant train --wandb logs the same per-epoch metrics already written to
metrics.csv, so wandb stays an optional extra (`uv sync --extra wandb`)
that nothing else depends on. A run's id is derived from the checkpoint
out_dir so --resume reattaches to the existing run instead of starting
a new one.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Swap the local/editable ETPlot `gallery[plotting]` dependency for
`plotstyle>=1.0.0`, now published to a package registry, so the repo
doesn't need a local ETPlot checkout to resolve.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth
diagnostics) with a lean giant/analysis/ package that compares one
autoregressive `giant rollout` for a checkpoint against a held-out
miniCaloSim reference file, and generates publication-styled plots in
parallel on HTCondor.
Rollout output and a raw reference file share a world-frame physical
column subset under identical names, so the old ALR/local-frame decode
machinery is gone — everything is world-frame mm/MeV.
- sources.py: canonical LazyFrames, synthetic-termination-row filtering,
the secondary view (rollout generation>0 tracks vs reference sec_*_list).
- reduce.py: streaming primitives — a single hist1d group_by pass, per-event
scalars, edep-weighted depth/transverse profiles, species share, leakage.
- context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material
group sets once into shared.json, so each compute job is one pass, no range
scan (histogram efficiency).
- catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg,
material}, per-event totals, shower profiles, species/leakage, secondaries.
- render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata.
- condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit):
one job per plot, compute/render split (workers polars-only, no LaTeX).
Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover
the reduce primitives, catalog id uniqueness + compute, condor submit, and a
guarded render smoke test. Delete the two predict-diagnostics notebooks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds model.conditioning = "physical" | "embedding": physical mode routes
particle mass/charge and material Z_eff/A_eff/density/X0/lambda_int through
small MLPs to replace the learned PDG/material embedding tables, so the
surrogate generalizes to PDG codes/materials outside the training vocab
instead of memorizing it. "embedding" stays available as the comparison
baseline (old checkpoints without the key default to it).
Stage 2 now regresses a secondary's mass/charge directly against a fixed
physics-derived target instead of a learned/snapped embedding, and uses no
snapping at inference — the model's raw predicted (mass, charge) is the
secondary's physical identity, including for its own further rollout steps.
A separate reporting-only nearest-known-PDG lookup (never fed back into the
model) populates output pdg columns / the embedding-mode rollout fallback.
giant/materials.py's table is populated with Geant4's own built-in NIST
constants (Z_eff, A_eff, density, X0, lambda_int), extracted directly from
the Geant4 11.4.1 build vendored in minicalosim via G4NistManager rather
than hand-typed literature values. G4_LYSO is left unfilled: confirmed (both
by runtime lookup and by searching minicalosim's history) that it's never
actually a constructed Geant4 material there, only documentation/UI color-map
text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the loop from single-step prediction into full showers:
- giant/geometry.py + `dwarf build-geometry-oracle`: learn position ->
(material, layer_id) from data (KNN/SVM) to supply the conditioning the
surrogate does not predict; flag detector escape by NN distance.
- giant/rollout.py: breadth-first batched frontier that steps all active
tracks, spawns secondaries as new tracks, and terminates on energy cutoff,
per-track max steps, escape, or natural end. Energy is deposited locally on
every stop except escape (leakage), so showers conserve energy exactly.
- `giant rollout` CLI: seed from real events (argmax pre_E), load checkpoint,
write a world-frame steps parquet + YAML sidecar.
- giant/analysis.py: compute_rollout_observables + plot_rollout_* for
single-sided longitudinal/transverse/total-energy shower profiles;
analysis/export_rollout_observables.py driver.
- scikit-learn added as an optional `geometry` extra (lazy-imported).
- Tests: tests/test_geometry.py, tests/test_rollout.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- energy_simplex_encode: warn when clipping post_E to pre_E discards
recorded edep/e_sec instead of silently zeroing them
- local/inv_local_frame_rotation: validate and normalize pre_dir instead
of silently assuming unit norm; raise on near-zero-norm rows
- train(): make --lr authoritative on resume instead of being silently
overwritten by the checkpoint's optimizer/scheduler state; print and
exit cleanly instead of silently training zero epochs when the
checkpoint already meets --epochs; truncate metrics.csv on a fresh
run instead of always appending
- dwarf update-manifest: check file existence for every manifest line,
not just ones whose gen/schema actually changed
- pyproject.toml: dev extra now pulls in convert+analysis so the
documented `uv sync --extra cpu --extra dev` + `pytest` actually
passes collection
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
When no --out is given, predictions are written to a UUID-named parquet
(/ceph central store for ceph inputs, sibling directory otherwise) and a
YAML reference file is created next to the checkpoint recording the output
path, dataset, checkpoint, and timestamp. Adds pyyaml as a core dependency
and unit tests for the two new helper functions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Move ipykernel into the analysis extra instead of a separate
dependency group, since it's needed wherever analysis plotting runs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ruff removed unused imports across analysis.py and several test files.
ty caught a wrong dict[int, int] annotation on StreamingStepsDataset's
mat_map (materials are strings) and a real bug in steps_to_parquet.py
where --compression none passed None to polars' write_parquet, which
only accepts the literal "uncompressed". Also narrows a few
Optional-typed attributes (ddpm_schedule, Normalizer.mean/std) with
asserts and aligns __getitem__'s parameter name with torch's Dataset
base class.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Pins torch to 2.3.x via mutually-exclusive cpu/cuda uv extras (newer
torch requires newer NVIDIA drivers), and adds upper bounds to the
other dependencies based on current PyPI releases.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
load_predicted_local now reads predict parquet via a lazy polars scan with
column projection pushed into the reader, instead of materializing the
whole file as a pandas DataFrame. Also adds marginal_table_pl and
constraint_report_pl, polars-native duplicates that read straight from a
predict parquet path/LazyFrame and stay lazy per (group, dim) pair, so
peak memory is one column slice rather than the whole SampleCollection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Provides stratified marginal comparisons, joint-structure checks (correlation
matrices, physically-coupled pairwise plots, direction alignment), and
physical-constraint validation (unit-norm directions, non-negative raw
targets) for a trained model's generated samples, building on the aggregate
marginal/KL check already in giant.validate.
Supports two entry points: live sampling against a checkpoint + val data
(load_model_bundle/collect_samples), or loading a precomputed
`giant predict --coord local` parquet directly (load_predicted_local) without
needing the checkpoint at all. Predict output is now tagged with parquet
schema metadata so the loader can verify a file's format and reject
coord=global or untagged files with a clear error instead of guessing from
column names.
Also extends the config git-hash mismatch warning (added for --config
loading) to checkpoint loading: both `giant predict` and
analysis.load_model_bundle now look for a config.toml next to the checkpoint
and warn (without failing) if it was generated from a different git commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Streaming pipeline: row-group-level parquet reading (PyArrow) so
large files never fully land in RAM; Welford online algorithm for
normalizer fitting; StreamingStepsDataset with shuffle buffer and
multi-worker file striping; event-ID scan and vocab scan via cheap
single-column reads
- giant/cli.py: typer-based CLI with `giant train` subcommand, mirroring
scripts/train.py; --shuffle-buffer flag for RAM control
- pyproject.toml: add typer>=0.12 dependency and giant entry point
- train.py: replace len(loader.dataset) with local counters (compatible
with IterableDataset which has no __len__)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
lars