Prototype a mixture-of-experts / routing-tree architecture #4

New Issue

2026-07-08T13:52:19+02:00

lbogner commented

2026-07-08 13:52:19 +02:00

(Migrated from gitlab.etp.kit.edu)

Prototype a mixture-of-experts / routing-tree architecture — the direction Jan and Tobias were most interested in.

This is not a full soft-gated MoE. It's a tree of several small networks, with one selected per call based on the condition, so each call evaluates one small expert instead of a single large monolithic net — the source of the speedup against the ~10× native-Geant4 budget.

Routing conditions (individually or combined): PDG (species), energy (regime), process (interaction type).

Differentiability: a hard branch on a continuous variable (energy) breaks gradients at the boundary. For the energy split use a soft turn-on — a stochastic/soft gate rather than a hard cut — so routing stays differentiable. Discrete pdg/process selectors don't have the boundary-gradient issue. This is the same problem class as the n_sec multiplicity gradient (Gumbel-Softmax / straight-through), applied to the router. Compare per-call cost and quality against the monolithic Phase 2 baseline.

Source: Jan & Tobias meeting 2026-07-07 §3 (incl. the soft-turn-on action item).

Prototype a **mixture-of-experts / routing-tree** architecture — the direction Jan and Tobias were most interested in. This is **not** a full soft-gated MoE. It's a tree of **several small networks**, with **one selected per call** based on the condition, so each call evaluates one small expert instead of a single large monolithic net — the source of the speedup against the ~10× native-Geant4 budget. **Routing conditions** (individually or combined): PDG (species), energy (regime), process (interaction type). **Differentiability:** a hard branch on a continuous variable (energy) breaks gradients at the boundary. For the energy split use a **soft turn-on** — a stochastic/soft gate rather than a hard cut — so routing stays differentiable. Discrete pdg/process selectors don't have the boundary-gradient issue. This is the same problem class as the `n_sec` multiplicity gradient (Gumbel-Softmax / straight-through), applied to the router. Compare per-call cost and quality against the monolithic Phase 2 baseline. Source: Jan & Tobias meeting 2026-07-07 §3 (incl. the soft-turn-on action item).

lbogner commented

2026-07-08 14:43:53 +02:00

(Migrated from gitlab.etp.kit.edu)

created branch 4-prototype-a-mixture-of-experts-routing-tree-architecture to address this issue

created branch [`4-prototype-a-mixture-of-experts-routing-tree-architecture`](/lbogner/giant/-/compare/master...4-prototype-a-mixture-of-experts-routing-tree-architecture) to address this issue

lbogner commented

2026-07-17 12:31:05 +02:00

(Migrated from gitlab.etp.kit.edu)

created branch 4-prototype-a-mixture-of-experts-routing-tree-architecture-2 to address this issue

created branch [`4-prototype-a-mixture-of-experts-routing-tree-architecture-2`](/lbogner/giant/-/compare/master...4-prototype-a-mixture-of-experts-routing-tree-architecture-2) to address this issue

lbogner commented

2026-07-17 12:31:14 +02:00

(Migrated from gitlab.etp.kit.edu)

mentioned in merge request !2

lbogner commented

2026-07-17 12:32:21 +02:00

(Migrated from gitlab.etp.kit.edu)

mentioned in merge request !3

lbogner commented

2026-07-17 12:32:47 +02:00

(Migrated from gitlab.etp.kit.edu)

mentioned in commit 098dacb8cd

mentioned in commit 098dacb8cdd0cf555ba588b7029c30aadc2ba8f6

lbogner (Migrated from gitlab.etp.kit.edu) closed this issue

2026-07-17 12:32:48 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#4