Counting Bytes and Bubbles: Kernels for Fast Mixture-of-Experts Decoding on a Single GPU

Generating text with a large language model is mostly a waiting game for the GPU: for every new token,the model’s weights and the attention memory must be streamed in from memory, while the arithmetic unitssit mostly idle. We describe a set of GPU kernels that we built for the Muse Spark Open Kernel Challenge,which times one layer of a mixture-of-experts model against the production SGLang serving engine on anNVIDIA H200. Our kernels make the whole decoding layer 1.79× faster than SGLang (geometric meanover six realistic serving loads), the routed-expert block 2.04×, attention 1.85×, dense projections 1.33×and the prompt-processing (prefill) layer 1.57×, while keeping the model’s weights and numerics unchanged.The improvements come from a few simple ideas: storing 16-bit weights in 12 or 13 bits without losingany information, splitting long attention work into many small pieces and merging them in a tree, treatingpopular and rare experts differently, chaining kernels so the next one starts loading before the previous onefinishes, and choosing among variants only after repeated measurement. We also report what did not work,which was most of what we tried.