lars/giant
1
0
Fork 0
Code Issues 4 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Add mixture-of-experts routing prototype for Stage 1 and Stage 2 #11

Merged
lbogner merged 12 commits from 4-prototype-a-mixture-of-experts-routing-tree-architecture into master 2026-07-17 12:32:47 +02:00
Conversation 1 Commits 12 Files Changed 16
+2152 -160
lbogner commented 2026-07-17 12:32:17 +02:00 (Migrated from gitlab.etp.kit.edu)
Copy Link
Copy Source

Both stages can now route through a pluggable Router (EnergyRouter as the
first implementation, a soft turn-on gate over pre-step log-energy) into
several small ExpertTrunks instead of one monolithic trunk. Trains as a
differentiable soft mixture and dispatches to a single expert per row at
eval time, which is the source of the per-call speedup this prototype is
after (issue #5's ~10x native-Geant4 budget). Disabled by default, so
existing configs/checkpoints are unaffected; build_models() centralizes
routed-vs-monolith construction across train/predict/rollout.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

Closes #4

Both stages can now route through a pluggable Router (EnergyRouter as the first implementation, a soft turn-on gate over pre-step log-energy) into several small ExpertTrunks instead of one monolithic trunk. Trains as a differentiable soft mixture and dispatches to a single expert per row at eval time, which is the source of the per-call speedup this prototype is after (issue #5's ~10x native-Geant4 budget). Disabled by default, so existing configs/checkpoints are unaffected; build_models() centralizes routed-vs-monolith construction across train/predict/rollout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Closes #4
lbogner commented 2026-07-17 12:32:47 +02:00 (Migrated from gitlab.etp.kit.edu)
Copy Link
Copy Source

mentioned in commit 098dacb8cd

mentioned in commit 098dacb8cdd0cf555ba588b7029c30aadc2ba8f6
lbogner (Migrated from gitlab.etp.kit.edu) merged commit 098dacb8cd into master 2026-07-17 12:32:47 +02:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
architecture
experiment
investigation
performance
phase2
validation
No Label architecture
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
lars (Lars Bogner)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: lars/giant#11
Delete Branch "4-prototype-a-mixture-of-experts-routing-tree-architecture"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?