Add learnable per-expert width and shared temperature to EnergyRouter #25

Merged

lars merged 2 commits from feat/router-learnable-width into master 2026-07-30 16:18:35 +02:00

Conversation 0 Commits 2 Files Changed 5

+304 -18

lars commented 2026-07-30 15:35:54 +02:00

Owner

Copy Link

Copy Source

EnergyRouter's gate sharpness was a single fixed temperature shared by
every expert, with no way for an expert to independently learn how much
of the energy axis it covers. Adds two mutually exclusive, default-off
modes: learn_width (per-expert learnable width) and learn_temperature
(single learnable shared scalar), both bounded via a sigmoid
interpolation warm-started to reproduce today's fixed-temperature gate
exactly at init, to compare against each other without risking the
unbounded-width collapse failure mode. Also promotes gate_stats's
entropy into a generic, optional Router.entropy_loss (lambda_entropy) as
a secondary guard against all experts' widths co-inflating together.

Co-Authored-By: Claude Sonnet 5 noreply@anthropic.com

EnergyRouter's gate sharpness was a single fixed temperature shared by every expert, with no way for an expert to independently learn how much of the energy axis it covers. Adds two mutually exclusive, default-off modes: learn_width (per-expert learnable width) and learn_temperature (single learnable shared scalar), both bounded via a sigmoid interpolation warm-started to reproduce today's fixed-temperature gate exactly at init, to compare against each other without risking the unbounded-width collapse failure mode. Also promotes gate_stats's entropy into a generic, optional Router.entropy_loss (lambda_entropy) as a secondary guard against all experts' widths co-inflating together. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-30 15:35:55 +02:00

Add learnable per-expert width and shared temperature to EnergyRouter ... da5f54ea1c

EnergyRouter's gate sharpness was a single fixed temperature shared by
every expert, with no way for an expert to independently learn how much
of the energy axis it covers. Adds two mutually exclusive, default-off
modes: learn_width (per-expert learnable width) and learn_temperature
(single learnable shared scalar), both bounded via a sigmoid
interpolation warm-started to reproduce today's fixed-temperature gate
exactly at init, to compare against each other without risking the
unbounded-width collapse failure mode. Also promotes gate_stats's
entropy into a generic, optional Router.entropy_loss (lambda_entropy) as
a secondary guard against all experts' widths co-inflating together.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

lars added 1 commit 2026-07-30 15:58:57 +02:00

Merge branch 'master' into feat/router-learnable-width aa55c407ab

lars merged commit 43cb6dd9ae into master 2026-07-30 16:18:35 +02:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

architecture

experiment

investigation

performance

phase2

validation

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

lars (Lars Bogner)

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lars/giant#25