Foundations of Sparse Mixture-of-Experts Routing
Modern Mixture-of-Experts (MoE) architectures achieve substantial parameter scaling without incurring proportional compute costs by sparsely routing tokens to specialized subnetworks. However, standard training regimes frequently suffer from structural imbalances and routing collapse, driven by greedy short-term utility maximization. This technical note serves as the foundational first installment in a series establishing a rigorous control-theoretic and optimization-based framework for MoE routing. Rather than relying on heuristic penalty terms, we formulate expert load balancing as a Stochastic Resource Allocation Program over an infinite horizon, bridging the ideal mathematical problem formulation with the empirical constraints of mini-batch deep learning training loops.
Authors
- Carl Johan Rehn (ORCID: https://orcid.org/0009-0002-7844-0770)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24284690
- Primary Topic
- Advanced Neural Network Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00