Mixture-of-Quants: Hosting High-Precision LLMs at Low-Precision Memory Bandwidth

Serving a Large Language Model (LLM) to a single user is bound by memory bandwidth: every generated token requires the entire weight set to be read from memory. Quantising weights to 4 bits halves the traffic of an 8-bit model but measurably degrades reasoning. We propose Mixture-of-Quants (MoQ), which exploits the observation that at each decode step the difference between the 4-bit model's two highest token probabilities indicates whether the 8-bit weights would produce a different output. By default, MoQ decodes each new token at 4 bits. Only when the top-2 softmax margin falls below a threshold is the token recomputed from the 8-bit weights. The 8-bit weights are stored as the 4-bit plane plus a 4-bit residual, so the store is the same size as an 8-bit model but only 4 bits per weight are read on the majority of tokens. Across three 27B–31B parameter architectures and \(\approx\) 300,000 scored positions, the margin predicts 4-bit/8-bit disagreement with AUROC of 0.93–0.95, catching 80% of disagreements whilst escalating 12%–16% of positions. On 3,000 paired SuperGPQA questions with a 32,768-token reasoning budget, escalating 31% of tokens lifts accuracy from 57.2% (4-bit) to 58.9%, matching the 8-bit model (58.9%, exact McNemar \(p = 1.000\)) and beating 4-bit at the same significance as the 4-bit/8-bit gap itself (\(p = 0.010\)). By physically relocating weights between HBM3 (3,850 GB/s) and Grace LPDDR (84 GB/s) on GH200 superchips, we show MoQ's throughput advantage over 8-bit rises monotonically from 1.38\(\times\) to 1.70\(\times\) as bandwidth falls, and that placing the rarely-read residual plane in slow memory is 3.9\(\times\) faster than spreading the same bytes uniformly, a trade a flat 8-bit model cannot make.