Implement n_sec.mode = "stop_token" (per-token EOS for the AR secondary decoder) #40

New Issue

2026-08-13T15:08:01+02:00

lars commented

2026-08-13 15:08:01 +02:00

The single most "v0.3" thing missing from the v0.3 branch.

Stage 2 is now autoregressive, but multiplicity is still predicted the v0.2
way: a one-shot classifier over {0..K_MAX} reading conditioning alone (plus
the stage-1 context), run before any token is generated; the AR loop then always
executes k_max slots and throws the tail away (sample_secondaries_ar,
giant/sample.py:297 — for k in range(k_max), unconditionally).

So the one-shot assumption survives inside the autoregressive decoder. A real
EOS token — one extra stop logit per token, sampled per slot, terminating the
loop — would:

make the length distribution conditional on the generated prefix, which is
the entire point of going autoregressive (a 12th secondary's existence depends
on what the first 11 took from the energy budget; n_sec_head cannot see that)
remove K_MAX = 15 as a modeling ceiling, leaving it as a safety cap
cut inference cost from a fixed k_max × steps model calls per physics step
to n_sec × steps — on a distribution where most steps produce 0–2
secondaries, that's most of stage 2's rollout cost

Everything needed is in place: the schema key exists (n_sec.mode = "stop_token", currently rejected as unimplemented at config.py:1252), the AR
loop is written, the prefix mask (StageTrainer._sec_mask) and the per-token
loss already exist. The work is a stop head, a per-token BCE term with the same
mask, and a break condition in the sampler.

Migrated from issues.md (v0.3.0 branch review, 2026-08-13), Issue 15.

**The single most "v0.3" thing missing from the v0.3 branch.** Stage 2 is now autoregressive, but *multiplicity* is still predicted the v0.2 way: a one-shot classifier over `{0..K_MAX}` reading conditioning alone (plus the stage-1 context), run before any token is generated; the AR loop then always executes `k_max` slots and throws the tail away (`sample_secondaries_ar`, `giant/sample.py:297` — `for k in range(k_max)`, unconditionally). So the one-shot assumption survives *inside* the autoregressive decoder. A real EOS token — one extra stop logit per token, sampled per slot, terminating the loop — would: - make the length distribution conditional on the **generated prefix**, which is the entire point of going autoregressive (a 12th secondary's existence depends on what the first 11 took from the energy budget; `n_sec_head` cannot see that) - remove `K_MAX = 15` as a *modeling* ceiling, leaving it as a safety cap - cut inference cost from a fixed `k_max × steps` model calls per physics step to `n_sec × steps` — on a distribution where most steps produce 0–2 secondaries, that's most of stage 2's rollout cost Everything needed is in place: the schema key exists (`n_sec.mode = "stop_token"`, currently rejected as unimplemented at `config.py:1252`), the AR loop is written, the prefix mask (`StageTrainer._sec_mask`) and the per-token loss already exist. The work is a stop head, a per-token BCE term with the same mask, and a `break` condition in the sampler. --- Migrated from `issues.md` (v0.3.0 branch review, 2026-08-13), Issue 15.

lars added the feature label 2026-08-13 15:08:01 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#40