Every merge to master previously produced three separate commits (bump
version -> update changelog, tagged here -> update README badges), so
the published tag never carried the current release's own README
badges, and the badge commit leaked into the next release's changelog
since no cliff.toml parser skipped it.
- Extract the bump/changelog/badge assembly into
.gitea/scripts/release-commit.sh, used by both the merge-to-master
path and the hand-pushed-tag sync path, so every tag now points at
one complete "chore: release vX.Y.Z" commit. Push the commit and its
tag atomically.
- sync-version-on-tag now refuses to touch a tag whose commit isn't
reachable from master (rather than silently rewriting an unreviewed
tree), and builds a proper release commit via the same script when
it does need to correct a hand-pushed tag's version.
- publish-package now depends only on sync-version-on-tag: since a tag
can only pass that guard if its commit is already on master, and
master is always fully checked, re-running the lint/type/test matrix
on tag pushes was redundant.
- Factor the repeated checkout/setup-uv/env/sync steps into a local
composite action (.gitea/actions/setup), fix `test`'s `needs` to
include ruff-format, and bump actions/upload-artifact to v4.
- cliff.toml: skip "chore: release ..." commits from the changelog.
Verified by dry-running release-commit.sh against a scratch worktree
for all three code paths (patch bump, --no-bump, explicit VERSION
sync), confirming idempotency and that the resulting commit carries
pyproject.toml, .bumpversion.toml, uv.lock, CHANGELOG.md and README.md
together.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NnotyatakKNS4NLjDbfYw1
sample_secondaries_ar ran all k_max=15 slots for every row regardless of
each row's own predicted secondary count, even though the baseline
checkpoint's rollout measured only 0.382 secondaries/step — so ~97% of
stage-2 model calls generated tokens sec_valid then masked away.
Compact the loop to the still-active row set at each slot: drop a row the
moment its n_sec_pred is exhausted (or, under n_sec.mode="stop_token", the
moment its own stop logit fires), so slot k's model calls cost O(active
rows) instead of O(B). Exact — rows are independent given their own
history — verified by comparing the compacted path against a new
full_length=True escape hatch that reproduces the original uncompacted
behavior bit-for-bit under deterministic noise.
full_length=True is required by
_assemble_stage2_ar_inputs_scheduled's scheduled-sampling self-sample,
whose training contract needs a real prediction at every slot up to
k_max regardless of a row's own count, so training behavior is
unchanged.
AttentionHistory's KV cache and MarkovHistory's O(1) state are kept
aligned to the shrinking active set via a new
HistoryEncoder.select_cache / Stage2Autoregressive.select_history_cache.
Also fixes a latent bug the refactor surfaced: derived_n_sec (stop-token
mode) could be overwritten by a later spurious re-fire of the stop logit
on a row that had already stopped; now tracked via an explicit `finished`
mask so only the first stop slot is recorded, matching the documented
contract.
No architecture or checkpoint-format change — every existing v0.3.0
Stage2Autoregressive checkpoint (flow/wgan, markov/attention,
head/stop_token) picks up the speedup automatically on its next
`giant rollout`/`giant predict`, no retraining needed.
Measured (CPU, hidden_dim=512/6 blocks, k_max=15, batch 512, mean
n_sec≈0.38 matching the baseline checkpoint's own rollout): 17.6-22.9x
fewer wall-clock seconds for the AR loop alone (attention/markov history
respectively). Directional only — baseline.toml's GPU inference-cost
comment is updated accordingly, flagged stale pending a real rollout
re-measurement via eval_cost_per_step.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPt7bVLZYFJe5cG6V7ahqC
Rewrite README.md from scratch as a scannable landing page (hero, one
mermaid pipeline diagram, quick start, deep detail folded into
collapsible sections) instead of the old flat prose dump duplicating
CLAUDE.md.
Swap the static "CI" badge for a live Gitea Actions status badge, and
add an update-badges job to ci.yml that recomputes the version and
test-count badges on every push to master and pushes an update only
when they actually changed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JL1hFbhLv5uwjTXqkWTLnH
giant.pipeline.run_setup_stage (used by both giant train and dwarf
warm-cache) previously opened and fully read each parquet file 4-6
separate times via pandas, with per-row Python loops padding the
secondary list columns on every chunk of the normalizer-fitting pass.
- giant/data/loader.py: pandas -> polars throughout; ragged sec_*_list
padding is now a single vectorized polars expression instead of a
per-row Python loop (including a .iloc[i] loop for directions).
- giant/data/scan.py (new): a fused metadata scan answering the event
index, pdg/material vocab, process counts, and pooled-pdg counts in
one pass per file instead of one pass per section. Frequency-ranking
ties are now an explicit (-count, first_seen) contract instead of an
accident of pandas' value_counts iteration order.
- giant/pipeline.py: run_setup_stage restructured to consult the cache
for every section first, then issue one combined scan request for
whatever's missing.
- giant/geometry.py: ported the one remaining pandas groupby to polars.
- pyproject.toml: polars promoted to a core dependency, pandas moved
to dev (only test fixtures still use it).
- giant/tools/profile_setup_scan.py (new): synthetic-data benchmark
for this scan, mirroring profile_analysis_costs.py's pattern.
Also fixes a real deadlock this surfaced: DataLoader worker
subprocesses fork() on Linux, and polars' native thread pool doesn't
survive a fork — a worker touching polars after the parent already had
hangs instantly. giant/pipeline.py's train/val DataLoaders now use
multiprocessing_context="spawn" whenever num_workers>0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DdT32YWNEwnVLZUHsgdeSC
Branches off configs/baseline.toml with both stages on WGAN-GP and
stage-2 n_sec.mode = stop_token, 30 epochs — combines two unbenchmarked
roadmap axes (post-v0.3.0 WGAN, and the AR stop-token multiplicity mode)
into one variant that stays a single edit away from baseline for
attribution.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KzPghrmFcAJYrWUvHApY9N
Closes the roadmap's long-standing "no eval-latency number exists for any
configuration" gap. Instruments `giant rollout` to record per-physical-step
wall-clock cost in its YAML sidecar, adds a measured Geant4/miniCaloSim
per-step reference (giant/analysis/geant4_reference.py, from a 3-energy,
4-event-count-per-energy local benchmark), and wires both into a new
eval_cost_per_step PlotSpec in the giant analyze gallery.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Replaces the extrapolated pre-v0.3 weak-spot claims in baseline.toml's
header (which had the wrong sign on step-count error) with measured
numbers from the first full rollout validation of this exact config,
and adds a matching Roadmap entry in CLAUDE.md.
torch/pandas/pyarrow/polars/uproot/awkward/particle were all imported
at module scope in giant/cli.py and giant/tools/dwarf.py, so even
`--help` paid ~1.6-1.9s of import cost. Move those imports into the
command bodies that actually need them (following the deferred-import
pattern already used for analysis/render/plots/sklearn/wandb), cutting
`giant --help` to ~0.3s and `dwarf --help` to ~0.2s with no change to
any command's actual behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A bare "pull_request:" key parses to null in YAML, which the runner
apparently doesn't treat as "trigger with defaults" the way an empty
mapping does — the open PR for this branch got no CI run at all.
- Drop [skip ci] from the bump-version/changelog/tag-sync commits so
master's tip always has a check run instead of only the merge commit.
- Filter those chore commits out of the changelog via message pattern
instead of the now-removed [skip ci] tag.
- Only run CI on push to master (plus tags); pull requests to any branch
still run the full suite.
- Add a publish-package job that builds and publishes to the Gitea PyPI
registry on every tag push, after tests and version sync pass.
giant predict/rollout rebuilt models straight from ckpt["model_config"] with
no way to change sampling-only keys (e.g. stage2_model.n_sec.stop_sampling)
without retraining. Adds config_overrides to load_for_inference, validated
against giant.config.INFERENCE_OVERRIDES so a typo or shape-bearing key
raises CheckpointCompatibilityError up front instead of an opaque
load_state_dict mismatch. Wired as a repeatable --set dotted.path=value on
both CLI commands, recorded in the rollout YAML sidecar, and surfaced in
`giant model summary`'s output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Taking argmax over the n_sec classifier logits collapses secondary
multiplicity onto its conditional mode at fixed pre-step conditioning,
under-dispersing n_sec in rollouts and biasing low wherever the true
conditional count distribution is right-skewed (typical for
multiplicity).
Generalizes stage2_model.n_sec.stop_sampling (previously stop_token-only)
into stage2_model.n_sec.sampling, covering both "head" (greedy: argmax;
sample: categorical draw via torch.multinomial) and "stop_token" (unchanged:
greedy threshold / Bernoulli draw) modes. stop_sampling is kept as a
deprecated alias in NSecConfig.from_dict and migrate_config, since it
appears in existing checkpoints' model_config. Default stays "greedy" so
existing runs/checkpoints are unaffected.
Replace the event-level n_sec confusion matrix with two step-resolved
secondary-multiplicity comparisons:
- sec_count_per_step: overlay histogram of how many secondaries a single
step emits, rollout series vs reference.
- sec_count_per_step_by_species: heatmap of per-step multiplicity of one
species (zero row included) against species, drawn as one panel per
rollout plus a reference panel, raw counts on a log color scale.
Both are backed by a new sources.secondaries_by_step view, which tags each
secondary with its emitting step — (event_id, parent_id, birth position)
on the rollout side, the row index on the reference side — so neither plot
needs a join against the step frame. Steps that emitted nothing are
recovered by subtraction from the chunk's step count, keeping both specs
sum-mergeable across condor chunks.
The rollout multiplicity is derived from the actual secondary birth rows
rather than the n_sec_pred column, which records the predicted count
before the per-event max-tracks cap.
_render_heatmap gained reference-panel and log-color support;
marginal_distance_summary sets neither key and is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CLAUDE.md still described the pre-v0.3.0 codebase: the Stage-2
autoregressive redesign as "designed, not implemented", a monolithic
network.py, a single global model.conditioning switch, and WGAN as
"implemented, not yet tested".
- Architecture rewritten around the actual giant/model split
(layers/encoders/trunks/routers/history/objectives/models/builders/
_legacy/summary; network.py is now a re-export shim), plus
cond_layout.py, checkpoint_io.py, _migration.py, data/setup_cache.py
and giant/training/.
- Conditioning documented per axis (conditioning.particle /
conditioning.material, each physical|embedding|onehot, freely mixed).
- Stage 2 documented with both decoders, n_sec.mode, teacher forcing,
stage1_context and the three particle_type.target options.
- Roadmap: v0.3.0 recorded as implemented/released; WGAN and MoE routing
as implemented but unvalidated, with the router retrain as next step.
- Analysis: run dir is <cwd>/analysis_runs/analysis_<id>, plus
variables/reduced/runtime_estimate and analyze list/merge-one/metrics.
- Added giant model summary, configs/, and the CI-automated version and
changelog bump.
README drift fixes only: project tree for the model/analysis/training
splits, analyze run-dir default, missing subcommands, --precision and
--stage2-stage1-context, the extras list, and two accuracy fixes
(--router configures stage 1 only; --conditioning sets two independent
axes at once).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The automated changelog (gitea #50) deliberately started fresh with no
backfill; this reverses that call now that it's wanted. v0.2.0-v0.3.2 are
generated from tag history via git-cliff/cliff.toml, matching the format of
existing entries. v0.3.3 was bumped but never tagged, so its commits stay
folded into the existing v0.3.4 entry. The v0.2.0 range (198 uncurated
pre-automation commits) is hand-curated to drop duplicate commits and
dev-log noise (WIP markers, incomplete-validation runs, repeated
"Apply ruff format").
shower_containment_depth_90/95's title contains a literal "%" (e.g.
"...(90% of deposited energy)"), which usetex reads as a comment marker
and aborts LaTeX compilation. Since render_all processes reduced JSON
files in sorted filename order, this killed every plot id sorting after
these two in the same run.
Escape title/xlabel once, centrally, in render()'s dispatch (the one
place every renderer kind draws them from before handing off to
plotstyle/matplotlib) rather than at each catalog.py call site, so any
future catalog title with a %, &, #, etc. is covered automatically.
_plot_metadata keeps using the unescaped Reduced for the gallery YAML,
since that's not LaTeX.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
giant analyze compares N rollout YAMLs against one shared reference file
(all must name the same dataset, checked up front) instead of exactly one
rollout vs one reference, rendering each rollout as its own colored series
against a single reference line/panel. Series names come from a repeated
--label flag, else the YAML stem, else "rollout" for a single YAML — a
single-rollout run keeps rendering identically to before this change.
Bundle now holds a name-keyed dict of rollout sides instead of one fixed
pair, every catalog compute_partial/finalize builds a Reduced.payload
keyed the same way ("series": {name: ...}, "reference": ... as the one
distinguished non-rollout entry), and every renderer draws N series (or
N panels, for the two heatmap-shaped specs and the router/type-embedding
diagnostics, which are inherently one-matrix/one-checkpoint per rollout)
against the reference's fixed dashed-ink style.
ci
lars