n_sec_head / type_head MLPs are hand-rolled five times #36

New Issue

2026-08-13T15:07:03+02:00

lars commented

2026-08-13 15:07:03 +02:00

nn.Sequential(Linear(cond_out_dim, hidden_dim // 2), SiLU(), Linear(...))
appears five times across models.py (lines 96, 198, 207, 374, 382) — the
n_sec_head and type_head of three stage classes. The // 2 is an
undocumented magic ratio and the depth is fixed at two layers.

Proposal: build_mlp_head(in_dim, out_dim, hidden, depth, act) in
layers.py, plus an optional [stageN_model.heads] block
(hidden_ratio, depth). Worth doing because n_sec accuracy and species
accuracy are both known weak spots, and both classifiers are currently
untunable independently of the trunk they hang off.

Migrated from issues.md (v0.3.0 branch review, 2026-08-13), Issue 11.

`nn.Sequential(Linear(cond_out_dim, hidden_dim // 2), SiLU(), Linear(...))` appears five times across `models.py` (lines 96, 198, 207, 374, 382) — the `n_sec_head` and `type_head` of three stage classes. The `// 2` is an undocumented magic ratio and the depth is fixed at two layers. **Proposal:** `build_mlp_head(in_dim, out_dim, hidden, depth, act)` in `layers.py`, plus an optional `[stageN_model.heads]` block (`hidden_ratio`, `depth`). Worth doing because `n_sec` accuracy and species accuracy are both known weak spots, and both classifiers are currently untunable independently of the trunk they hang off. --- Migrated from `issues.md` (v0.3.0 branch review, 2026-08-13), Issue 11.

lars added the modularity label 2026-08-13 15:07:03 +02:00

lars added a new dependency 2026-08-13 15:23:29 +02:00

#39 Stage1Model / Stage2OneShot / Stage2Autoregressive duplicate ~80 lines of scaffolding

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Blocks

#39 Stage1Model / Stage2OneShot / Stage2Autoregressive duplicate ~80 lines of scaffolding

lars/giant

Reference: lars/giant#36