perf: compact Stage-2 AR inference loop to active rows only #94

2026-09-03T17:58:03+02:00

lars commented

2026-09-03 17:58:03 +02:00

sample_secondaries_ar ran all k_max=15 slots for every row regardless of
each row's own predicted secondary count, even though the baseline
checkpoint's rollout measured only 0.382 secondaries/step — so ~97% of
stage-2 model calls generated tokens sec_valid then masked away.

Compact the loop to the still-active row set at each slot: drop a row the
moment its n_sec_pred is exhausted (or, under n_sec.mode="stop_token", the
moment its own stop logit fires), so slot k's model calls cost O(active
rows) instead of O(B). Exact — rows are independent given their own
history — verified by comparing the compacted path against a new
full_length=True escape hatch that reproduces the original uncompacted
behavior bit-for-bit under deterministic noise.

full_length=True is required by
_assemble_stage2_ar_inputs_scheduled's scheduled-sampling self-sample,
whose training contract needs a real prediction at every slot up to
k_max regardless of a row's own count, so training behavior is
unchanged.

AttentionHistory's KV cache and MarkovHistory's O(1) state are kept
aligned to the shrinking active set via a new
HistoryEncoder.select_cache / Stage2Autoregressive.select_history_cache.

Also fixes a latent bug the refactor surfaced: derived_n_sec (stop-token
mode) could be overwritten by a later spurious re-fire of the stop logit
on a row that had already stopped; now tracked via an explicit finished
mask so only the first stop slot is recorded, matching the documented
contract.

No architecture or checkpoint-format change — every existing v0.3.0
Stage2Autoregressive checkpoint (flow/wgan, markov/attention,
head/stop_token) picks up the speedup automatically on its next
giant rollout/giant predict, no retraining needed.

Measured (CPU, hidden_dim=512/6 blocks, k_max=15, batch 512, mean
n_sec≈0.38 matching the baseline checkpoint's own rollout): 17.6-22.9x
fewer wall-clock seconds for the AR loop alone (attention/markov history
respectively). Directional only — baseline.toml's GPU inference-cost
comment is updated accordingly, flagged stale pending a real rollout
re-measurement via eval_cost_per_step.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_01HPt7bVLZYFJe5cG6V7ahqC

sample_secondaries_ar ran all k_max=15 slots for every row regardless of each row's own predicted secondary count, even though the baseline checkpoint's rollout measured only 0.382 secondaries/step — so ~97% of stage-2 model calls generated tokens sec_valid then masked away. Compact the loop to the still-active row set at each slot: drop a row the moment its n_sec_pred is exhausted (or, under n_sec.mode="stop_token", the moment its own stop logit fires), so slot k's model calls cost O(active rows) instead of O(B). Exact — rows are independent given their own history — verified by comparing the compacted path against a new full_length=True escape hatch that reproduces the original uncompacted behavior bit-for-bit under deterministic noise. full_length=True is required by _assemble_stage2_ar_inputs_scheduled's scheduled-sampling self-sample, whose training contract needs a real prediction at every slot up to k_max regardless of a row's own count, so training behavior is unchanged. AttentionHistory's KV cache and MarkovHistory's O(1) state are kept aligned to the shrinking active set via a new HistoryEncoder.select_cache / Stage2Autoregressive.select_history_cache. Also fixes a latent bug the refactor surfaced: derived_n_sec (stop-token mode) could be overwritten by a later spurious re-fire of the stop logit on a row that had already stopped; now tracked via an explicit `finished` mask so only the first stop slot is recorded, matching the documented contract. No architecture or checkpoint-format change — every existing v0.3.0 Stage2Autoregressive checkpoint (flow/wgan, markov/attention, head/stop_token) picks up the speedup automatically on its next `giant rollout`/`giant predict`, no retraining needed. Measured (CPU, hidden_dim=512/6 blocks, k_max=15, batch 512, mean n_sec≈0.38 matching the baseline checkpoint's own rollout): 17.6-22.9x fewer wall-clock seconds for the AR loop alone (attention/markov history respectively). Directional only — baseline.toml's GPU inference-cost comment is updated accordingly, flagged stale pending a real rollout re-measurement via eval_cost_per_step. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HPt7bVLZYFJe5cG6V7ahqC

lars added 1 commit 2026-09-03 17:58:04 +02:00

perf: compact Stage-2 AR inference loop to active rows only

CI / Sync project version with tag (pull_request) Has been skipped

Details

CI / Type check (ty) (pull_request) Successful in 1m2s

Details

CI / Lint (ruff check) (pull_request) Successful in 1m6s

Details

CI / Format (ruff format) (pull_request) Successful in 1m6s

Details

CI / Tests (pull_request) Successful in 2m46s

Details

CI / Bump version, tag, and update changelog on merge to master (pull_request) Has been skipped

Details

CI / Publish package to Gitea package registry (pull_request) Has been skipped

Details

CI / Update README badges (version, test count) (pull_request) Has been skipped

Details

5c576fa8f3

sample_secondaries_ar ran all k_max=15 slots for every row regardless of
each row's own predicted secondary count, even though the baseline
checkpoint's rollout measured only 0.382 secondaries/step — so ~97% of
stage-2 model calls generated tokens sec_valid then masked away.

Compact the loop to the still-active row set at each slot: drop a row the
moment its n_sec_pred is exhausted (or, under n_sec.mode="stop_token", the
moment its own stop logit fires), so slot k's model calls cost O(active
rows) instead of O(B). Exact — rows are independent given their own
history — verified by comparing the compacted path against a new
full_length=True escape hatch that reproduces the original uncompacted
behavior bit-for-bit under deterministic noise.

full_length=True is required by
_assemble_stage2_ar_inputs_scheduled's scheduled-sampling self-sample,
whose training contract needs a real prediction at every slot up to
k_max regardless of a row's own count, so training behavior is
unchanged.

AttentionHistory's KV cache and MarkovHistory's O(1) state are kept
aligned to the shrinking active set via a new
HistoryEncoder.select_cache / Stage2Autoregressive.select_history_cache.

Also fixes a latent bug the refactor surfaced: derived_n_sec (stop-token
mode) could be overwritten by a later spurious re-fire of the stop logit
on a row that had already stopped; now tracked via an explicit `finished`
mask so only the first stop slot is recorded, matching the documented
contract.

No architecture or checkpoint-format change — every existing v0.3.0
Stage2Autoregressive checkpoint (flow/wgan, markov/attention,
head/stop_token) picks up the speedup automatically on its next
`giant rollout`/`giant predict`, no retraining needed.

Measured (CPU, hidden_dim=512/6 blocks, k_max=15, batch 512, mean
n_sec≈0.38 matching the baseline checkpoint's own rollout): 17.6-22.9x
fewer wall-clock seconds for the AR loop alone (attention/markov history
respectively). Directional only — baseline.toml's GPU inference-cost
comment is updated accordingly, flagged stale pending a real rollout
re-measurement via eval_cost_per_step.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPt7bVLZYFJe5cG6V7ahqC

lars merged commit 9a03f4552a into master

2026-09-03 18:03:33 +02:00

lars deleted branch perf/stage2-ar-inference-compaction

2026-09-03 18:03:33 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#94