plotstyle is now a real installed package (from the git.larsbogner.de
index) with its own type info, so the unresolved-import suppression
is no longer needed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Swap the local/editable ETPlot `gallery[plotting]` dependency for
`plotstyle>=1.0.0`, now published to a package registry, so the repo
doesn't need a local ETPlot checkout to resolve.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`giant rollout` now records the checkpoint's architecture (mode,
hidden_dim, n_blocks, emb_dim, dropout, conditioning) plus training_epoch
and best_val_loss in its YAML sidecar, using data already loaded from the
checkpoint. condor.py carries those through run_meta, and render.py passes
them to plotstyle's new_figure(params=...) so every plot's subtitle shows
what produced it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
`giant analyze prep` / `submit` now take the `giant rollout` YAML sidecar as
their only positional input instead of explicit --rollout/--reference/--out-dir.
The YAML's `output`/`dataset` keys name the rollout parquet and its seed file
(the reference truth), and the rest of the sidecar (checkpoint, geometry oracle,
cutoffs) flows into every plot's gallery metadata.
prep derives its own run directory next to the rollout parquet
(<...>/analysis_<id>/) holding shared.json, run_meta.json, reduced/, plots/.
compute-one and render now take just --run-dir / a run-dir argument and read the
resolved paths + metadata from run_meta.json, so the condor wrapper no longer
threads file paths. open_side scans a directory of reference shards via glob.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Replace the monolithic giant/analysis.py (predict-local + RolloutVsTruth
diagnostics) with a lean giant/analysis/ package that compares one
autoregressive `giant rollout` for a checkpoint against a held-out
miniCaloSim reference file, and generates publication-styled plots in
parallel on HTCondor.
Rollout output and a raw reference file share a world-frame physical
column subset under identical names, so the old ALR/local-frame decode
machinery is gone — everything is world-frame mm/MeV.
- sources.py: canonical LazyFrames, synthetic-termination-row filtering,
the secondary view (rollout generation>0 tracks vs reference sec_*_list).
- reduce.py: streaming primitives — a single hist1d group_by pass, per-event
scalars, edep-weighted depth/transverse profiles, species share, leakage.
- context.py/grouping.py: prep resolves fixed bin edges + energy/pdg/material
group sets once into shared.json, so each compute job is one pass, no range
scan (histogram efficiency).
- catalog.py: declarative PlotSpec registry — marginals x {overall,energy,pdg,
material}, per-event totals, shower profiles, species/leakage, secondaries.
- render.py: the only plotstyle/LaTeX importer; PDFs + gallery metadata.
- condor.py + `giant analyze` CLI (prep/compute-one/list/render/submit):
one job per plot, compute/render split (workers polars-only, no LaTeX).
Styling via ETPlot's plotstyle (added to the analysis extra). New tests cover
the reduce primitives, catalog id uniqueness + compute, condor submit, and a
guarded render smoke test. Delete the two predict-diagnostics notebooks.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ruff check, ruff format check, ty check, and pytest now run as four
independent jobs instead of one sequential lint job gating test —
faster wall-clock CI since none of these checks depend on each other.
build still waits on all four before bumping/publishing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Unquoted on: gets parsed as the YAML 1.1 boolean true instead of the
string "on", so Gitea couldn't find a trigger key and didn't register
the workflow at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three-stage workflow: lint/format/type-check (ruff, ty), pytest,
and (master pushes only) a version-bump + uv build + publish to the
Gitea package registry. Bump commit is tagged [skip ci] to avoid
retriggering the workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
plot_router_gating visualizes soft expert gate weights vs. a continuous
routing axis (e.g. pre-step energy), binned into equal-population
quantiles and stacked to show the router's soft decision boundaries.
Wired into rollout_validation.ipynb as a new notebook-only section
that loads a checkpoint's Router directly, since gate weights aren't
present in rollout/predict parquet output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds --mode wgan alongside flow/ddpm: both stages get a WGAN-GP
generator/critic pair (giant.model.wgan) instead of flow matching, so
inference is a single forward pass per stage rather than a 10-step ODE
integration — the fast-eval architecture noted in the roadmap.
predict/rollout auto-detect the mode from the checkpoint's model_config.
Best-checkpoint selection for wgan uses marginal-KL against the EMA
generators every epoch, since a critic loss isn't a monotone quality
signal. --router is not supported together with --mode wgan.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- make_seed_frontier only resolves particle mass/charge in "physical"
mode, so "embedding"-mode rollouts no longer crash on a seed PDG code
giant.particles can't resolve (the TERM_UNKNOWN_PDG gate now handles it).
- nearest_known_pdg skips unresolvable candidate PDG codes instead of
raising and killing the whole rollout/predict run.
- predict/rollout fail with a clear message when a checkpoint predates
the sec_phys normalizer, instead of a bare KeyError.
- validate_marginals' phys_kl degrades to NaN (matching the
energy_fraction_kl pattern) instead of crashing when a validated batch
has zero secondaries on either side.
- Correct CLAUDE.md's stale claim that the materials table is unfilled.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Adds model.conditioning = "physical" | "embedding": physical mode routes
particle mass/charge and material Z_eff/A_eff/density/X0/lambda_int through
small MLPs to replace the learned PDG/material embedding tables, so the
surrogate generalizes to PDG codes/materials outside the training vocab
instead of memorizing it. "embedding" stays available as the comparison
baseline (old checkpoints without the key default to it).
Stage 2 now regresses a secondary's mass/charge directly against a fixed
physics-derived target instead of a learned/snapped embedding, and uses no
snapping at inference — the model's raw predicted (mass, charge) is the
secondary's physical identity, including for its own further rollout steps.
A separate reporting-only nearest-known-PDG lookup (never fed back into the
model) populates output pdg columns / the embedding-mode rollout fallback.
giant/materials.py's table is populated with Geant4's own built-in NIST
constants (Z_eff, A_eff, density, X0, lambda_int), extracted directly from
the Geant4 11.4.1 build vendored in minicalosim via G4NistManager rather
than hand-typed literature values. G4_LYSO is left unfilled: confirmed (both
by runtime lookup and by searching minicalosim's history) that it's never
actually a constructed Geant4 material there, only documentation/UI color-map
text.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Merged in every non-analysis change from the MoE-prototype branch (routing,
training, data pipeline, streaming rollout output), keeping this branch's
lean streaming giant/analysis.py and rebuilding the rollout-vs-truth feature
natively on it instead of resurrecting the old numpy SampleCollection path.
- Add RolloutVsTruth, accepted anywhere Tier 1-3 functions take a predict-parquet
source: decodes a giant rollout file and a held-out truth file into
RAW_TARGET_NAMES space via a polars port of the forward local-frame rotation,
fully streaming (no SampleCollection, no eager materialization).
- Add compute_rollout_vs_truth_observables_pl for Tier 4, reusing
EventObservables (now backed by independent real_table/gen_table to support
unequal rollout/truth event counts) so every existing shower-observable plot
function works unchanged for both one-step and full-rollout comparisons.
- Update analysis/rollout_validation.ipynb to the new API and CLAUDE.md's
architecture description; add test coverage for the new source type.
- Fix a pre-existing return-type mismatch in giant.rollout.rollout() (found by
`ty check`): the on_chunk summary-dict branch didn't match the declared
dict[str, np.ndarray] return type, now expressed as a RolloutSummary TypedDict.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Gives flow-matching sampling a cleaner EMA shadow copy to draw from (--ema-decay,
--weights raw|ema in predict/rollout), fixes the LR warmup/cosine schedule stepping
once per epoch even when an epoch is tens of thousands of steps, and caps the
per-epoch val-loss pass (--max-val-batches) so large val sets don't dominate epoch time.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Training runs the full soft mixture (every expert over the whole batch),
so routed activation memory scales with the expert count; the old estimate
used one expert's dims and would overshoot free VRAM by a factor of
n_experts. Fold the expert count into n_blocks for the training path
(inference's top-1 dispatch still just partitions the batch, so one
expert's dims bound it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Route on several independent axes at once (e.g. energy x pdg), each with
its own expert count and hyperparameters. The joint gate is the outer
product of per-axis softmax gates, so it stays a partition of unity and
top1/balance_loss factor per-axis. Config uses flat axis{i}_{field} keys
in model.router (TOML/CLI friendly), also settable via repeatable
--router-axis flags.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
_Recorder previously accumulated every generated step across all events/
tracks/steps in Python lists, materialised once at the end and written
via a single pq.write_table — memory scaled with n_events * max_steps *
avg_tracks_per_event. rollout() now takes an optional on_chunk callback
that streams each non-empty batch immediately (fixed per-key dtypes via
_RECORD_DTYPES keep every chunk's table schema identical, which
pq.ParquetWriter requires across writes); giant rollout wires this to an
incrementally-written ParquetWriter, mirroring the row-group streaming
giant predict already does on its input side. Without on_chunk, rollout()
keeps its old buffered return for existing callers/tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Routes on the pre-step PDG code, which — unlike ProcessRouter's process
label — is already known at gate time (a conditioning input), so no
supervision is needed and classify_loss falls back to the zero default.
Generalizes EnergyRouter's soft-turn-on-then-Voronoi trick from a 1-D
distance to a small learned PDG embedding space: its own embedding table
maps each PDG code to a point, and n_experts learnable centers partition
that space.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Brings in the rollout-validation fixes developed alongside Phase 2
(exact e_sec budget rescaling in decode_secondaries, filtering
synthetic termination rows out of load_rollout_vs_truth, Tier 4 truth
overlay, --energy-gev support in dwarf make-root) and reconciles them
with this branch's mixture-of-experts routing work: build_features/
build_models/dataset plumbing keep the ProcessRouter's proc_map/
proc_idx threading, and create_root_files.py's job_seed folds in both
the per-job seed derivation and the new energy_gev component.
run_pbwo4/run_sampling now accept a trailing energy_GeV positional arg;
thread it through plan/run/seed so datasets like pbwo4_10gev can be
generated at non-default beam energies.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
decode_secondaries's stick-breaking only guarantees valid secondary slots
sum to <= e_sec, leaving a shortfall that rollout.py silently dumped into
that step's edep. Rescale the valid slots by one common per-row factor
instead, so they sum to exactly e_sec whenever n_sec > 0: this spreads any
shortfall proportionally across all secondaries rather than concentrating
it in whichever slot is last by energy rank (which would let that one
low-energy secondary balloon and distort the shower's topology). Rows
where every valid slot decodes to ~zero fall back to an even split.
n_sec == 0 rows are unchanged (still nothing to carry the budget, so
rollout.py's edep top-up still applies there) — narrowed the related
caveat in load_rollout_vs_truth's docstring to just that case.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
load_rollout_vs_truth was including rollout.py's synthetic termination-
bookkeeping rows (escaped/unknown_pdg/energy_cutoff/max_steps) unfiltered:
these carry step_length=0 and edep=pre_E dumped in one row for shower-level
energy conservation, not a real per-step value, and nearly doubled the
apparent mean edep in a repro. _load_world_frame_side now drops them, keeping
only real generated steps (continuing or natural_end). Also documents the
remaining, unfixable difference: rollout's edep on real steps absorbs any
secondary-energy budget Stage 2 didn't allocate, which truth's edep never does.
Adds compute_truth_observables, the truth-schema counterpart to
compute_rollout_observables, so the Tier 4 event-level plots
(plot_rollout_longitudinal/transverse/total_energy) can overlay a real
reference computed directly from load_rollout_vs_truth's own truth file,
without needing a separate paired giant predict --coord local file. Shares
the depth/transverse binning core with compute_rollout_observables via a new
_event_axis_depth_transverse helper.
Updates rollout_validation.ipynb's Tier 4 section to use this reference and
points ROLLOUT_FILE/TRUTH_FILE at a real prediction/shard pair.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the Tier 1-3 SampleCollection diagnostics (marginals, correlations,
pairwise, direction alignment, constraints) to work on a full autoregressive
giant rollout shower checked against an independent ground-truth steps file,
rather than only paired giant predict --coord local output. The two files
are unpaired (different lengths, own conditioning), so SampleCollection
gains optional *_gen fields and _group_labels/marginal_table/plot_marginals/
plot_pairwise build independent real/gen masks instead of assuming one.
Adds analysis/rollout_validation.ipynb, a sibling of validation.ipynb built
around this workflow.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replace giant/analysis.py's dual numpy-SampleCollection + polars paths with a
single polars-streaming implementation that produces the validation notebook's
plots directly from a `giant predict --coord local` parquet, sized for files
larger than RAM.
- Drop the numpy SampleCollection path (load_predicted_local, marginal_table,
correlation_matrices, direction_alignment, constraint_report, plot_kl_bars)
and the rollout observables; the 5 remaining plotters now take a parquet
path / LazyFrame and stream internally.
- Rewrite compute_event_observables_pl to aggregate in parallel streaming
polars (post-pos reconstruction as expressions) instead of a serial
pyarrow-batch + numpy loop, fixing a pre-existing OOM (holistic median +
323M-row join in the bin-edge sizing). Medians are approximated from a
streaming log-bin histogram with within-bin interpolation.
- Keep every full-file scan narrow (few columns): on a file larger than RAM,
peak mmap memory, not scan count, is the binding constraint. Marginals run
one dim at a time (~15GB peak) rather than a combined all-dims pass (OOM).
- Update analysis/validation.ipynb to the path-based API; delete the
analysis/export_*.py and compare_ode_steps_*.py one-off scripts.
- Rewrite tests/test_analysis.py around parquet fixtures with an inline numpy
oracle; add correlation/streaming-plotter and approx-median coverage.
Verified end-to-end on the 32GB predict file: full notebook completes at
~25GB peak (no OOM); event rollup runs at ~13 cores.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A listed child_track_id can fail to match any first-step row (e.g. a
secondary absorbed below the tracking threshold at birth). The
parent->child left join in _add_secondary_attributes left these as
nulls, which silently became NaN once the parquet round-tripped
through the loader's float32 padding — poisoning every later secondary
slot in that step via the cumulative "remaining budget" in
encode_secondaries, while e_sec quietly undercounted and n_sec (from
len(child_track_ids)) overcounted relative to the actual lists.
Drop orphans from both the per-secondary lists and child_track_ids
itself so downstream counts stay consistent, and thread the per-file
orphaned count back through convert_steps_to_parquet so both the
sequential and --jobs>1 batch paths in `dwarf convert` can report an
aggregate total instead of relying on grepping printed output.
Also floors encode_secondaries' slot-0 budget to _EPS (matching the
i>0 branch), fixing a harmless but noisy 0/0 divide warning on
zero-secondary steps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A listed child_track_id can fail to match any first-step row (e.g. a
secondary absorbed below the tracking threshold at birth). The
parent->child left join in _add_secondary_attributes left these as
nulls, which silently became NaN once the parquet round-tripped
through the loader's float32 padding — poisoning every later secondary
slot in that step via the cumulative "remaining budget" in
encode_secondaries, while e_sec quietly undercounted and n_sec (from
len(child_track_ids)) overcounted relative to the actual lists.
Drop orphans from both the per-secondary lists and child_track_ids
itself so downstream counts stay consistent, and thread the per-file
orphaned count back through convert_steps_to_parquet so both the
sequential and --jobs>1 batch paths in `dwarf convert` can report an
aggregate total instead of relying on grepping printed output.
Also floors encode_secondaries' slot-0 budget to _EPS (matching the
i>0 branch), fixing a harmless but noisy 0/0 divide warning on
zero-secondary steps.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Concurrent job launches in create_root_files.py can start within the
same wall-clock second, and minicalosim's default seed falls back to
time(NULL) in that case — so two "independent" shards could silently
get identical RNG state and produce byte-identical physics. Requires
the companion MINICALOSIM_SEED env-var support in the minicalosim repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Concurrent job launches in create_root_files.py can start within the
same wall-clock second, and minicalosim's default seed falls back to
time(NULL) in that case — so two "independent" shards could silently
get identical RNG state and produce byte-identical physics. Requires
the companion MINICALOSIM_SEED env-var support in the minicalosim repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A parquet that carries child_track_ids/e_sec but was never run through the
parent->child join lacks the per-secondary columns (sec_E_list/sec_pdg_list/
sec_dir_list). build_features would fall back to all-zero sec_cont/sec_pdg_idx,
collapsing every secondary to PDG index 0 and a constant energy fraction — a
broken Stage 2 that trained with no error (single-species validation tables).
Add an opt-in require_secondaries flag that raises when n_sec > 0 but the lists
are absent, and enable it on the training paths (StreamingStepsDataset and the
normalizer-fit pass). giant predict keeps the default False for Stage-1-only use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A parquet that carries child_track_ids/e_sec but was never run through the
parent->child join lacks the per-secondary columns (sec_E_list/sec_pdg_list/
sec_dir_list). build_features would fall back to all-zero sec_cont/sec_pdg_idx,
collapsing every secondary to PDG index 0 and a constant energy fraction — a
broken Stage 2 that trained with no error (single-species validation tables).
Add an opt-in require_secondaries flag that raises when n_sec > 0 but the lists
are absent, and enable it on the training paths (StreamingStepsDataset and the
normalizer-fit pass). giant predict keeps the default False for Stage-1-only use.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Routes on the physics process (Compton, phot, brems, ...) that ends a
step, supervised by a small classifier since process is a post-step
outcome unobservable at gate time. Threads a process label end-to-end
through the data pipeline (loader, build_features, dataset batches,
training loss/checkpointing) alongside the existing EnergyRouter.
Both stages can now route through a pluggable Router (EnergyRouter as the
first implementation, a soft turn-on gate over pre-step log-energy) into
several small ExpertTrunks instead of one monolithic trunk. Trains as a
differentiable soft mixture and dispatches to a single expert per row at
eval time, which is the source of the per-call speedup this prototype is
after (issue #5's ~10x native-Geant4 budget). Disabled by default, so
existing configs/checkpoints are unaffected; build_models() centralizes
routed-vs-monolith construction across train/predict/rollout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Bring the docs in line with the current two-stage code: energy ALR
simplex output, 8D conditioning (n_sec/e_sec now predicted, not given),
the SecondaryDecoder stage, and the shower rollout + geometry oracle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
miniCaloSim's detector is a stack of planar layer slabs along one axis, so
material/layer_id are a pure function of depth. The new "slab" method
exploits this with an exact O(log #segments) binary search over
depth-axis segment boundaries, instead of a nearest-neighbour search over
hundreds of thousands of reference points — much cheaper per call, which
matters since the oracle is queried on every autoregressive rollout step.
"knn"/"svm" remain as fallbacks for non-slab geometries.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The estimate_batch_size(training=True) calibration point was measured on
the pre-Phase-2 architecture (hidden_dim=512). Re-measured against the
current hidden_dim=1024 stack (Stage-2 secondary decoder + n_sec head
included): ~29696 batch size at ~7683 MiB VRAM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The masked flow-matching loss for the secondary decoder averaged uniformly
over all 20 per-slot dims, letting the 16 type-embedding dims outvote the
4 physically-interesting ones (stick-break logit + direction). Split the
two blocks and average each over its own width before summing, so they
contribute with equal weight regardless of EMB_DIM.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Real data has steps with up to ~37 secondaries, but the n_sec head only
has K_MAX+1=16 classes. The unclamped label occasionally overflowed
cross_entropy's valid range and crashed CUDA training with
"unique_by_key: failed to synchronize: cudaErrorAssert". The
continuous secondary targets were already truncated to K_MAX slots;
only this label was missed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closes the loop from single-step prediction into full showers:
- giant/geometry.py + `dwarf build-geometry-oracle`: learn position ->
(material, layer_id) from data (KNN/SVM) to supply the conditioning the
surrogate does not predict; flag detector escape by NN distance.
- giant/rollout.py: breadth-first batched frontier that steps all active
tracks, spawns secondaries as new tracks, and terminates on energy cutoff,
per-track max steps, escape, or natural end. Energy is deposited locally on
every stop except escape (leakage), so showers conserve energy exactly.
- `giant rollout` CLI: seed from real events (argmax pre_E), load checkpoint,
write a world-frame steps parquet + YAML sidecar.
- giant/analysis.py: compute_rollout_observables + plot_rollout_* for
single-sided longitudinal/transverse/total-energy shower profiles;
analysis/export_rollout_observables.py driver.
- scikit-learn added as an optional `geometry` extra (lazy-imported).
- Tests: tests/test_geometry.py, tests/test_rollout.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
lars