Foundations of Sparse Mixture-of-Experts Routing

Modern Mixture-of-Experts (MoE) architectures achieve substantial parameter scaling without incurring proportional compute costs by sparsely routing tokens to specialized subnetworks. However, standard training regimes frequently suffer from structural imbalances and routing collapse, driven by greedy short-term utility maximization. This technical note serves as the foundational first installment in a series establishing a rigorous control-theoretic and optimization-based framework for MoE routing. Rather than relying on heuristic penalty terms, we formulate expert load balancing as a Stochastic Resource Allocation Program over an infinite horizon, bridging the ideal mathematical problem formulation with the empirical constraints of mini-batch deep learning training loops.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24284690
Primary Topic
Advanced Neural Network Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Foundations of Sparse Mixture-of-Experts Routing

Carl Johan Rehn
Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
article

Foundations of Sparse Mixture-of-Experts Routing

Carl Johan Rehn
article en

Abstract

Modern Mixture-of-Experts (MoE) architectures achieve substantial parameter scaling without incurring proportional compute costs by sparsely routing tokens to specialized subnetworks. However, standard training regimes frequently suffer from structural imbalances and routing collapse, driven by greedy short-term utility maximization. This technical note serves as the foundational first installment in a series establishing a rigorous control-theoretic and optimization-based framework for MoE routing. Rather than relying on heuristic penalty terms, we formulate expert load balancing as a Stochastic Resource Allocation Program over an infinite horizon, bridging the ideal mathematical problem formulation with the empirical constraints of mini-batch deep learning training loops.

Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon