Certified L∞-Bounded Compression for AI Models, II: Compressing a Language Model's Weights, Key–Value Cache and Activations

A language model keeps three kinds of numbers in memory: its weights, its key–value cacheand the activations that flow through it. This paper compresses all three under the certifiedL∞ contract of Part I, and shows what that does for inference. The weights of dense andmixture-of-experts models are stored as virtual-band indices (a 27B model in 6.22 bits perweight, 2.58× below its checkpoint). The key–value cache is stored under the same contractwith a per-channel grid, with every token, head and layer kept, at 6.8 bits per element on the27B model and perplexity within 5 × 10−4 of the dense cache. The activations of prefill arerounded per token to int8 with exact escape channels, an error bounded per call, and multipliedwith the weights’ own integer indices on int8 tensor cores, so that no second copy of the weightsis kept and perplexity changes by +0.04% on a 35B mixture-of-experts model. Because thepacked forms are executed directly, compression also speeds the model up: decoding reads fewerbytes, a tensor-core kernel verifies 8–16 speculative tokens at once with the checkpoint’s drafthead, n-gram lookup or the DFlash drafters, and int8 prefill is 1.2–1.4× faster on a dense 27Bmodel and 1.1–1.8× faster on a 35B mixture of experts. With the DFlash drafter and the int8prefill, the 27B dense model prefills at 937–1175 tokens/s and decodes at 13–16 tokens/s, andthe 35B mixture-of-experts model prefills at 1949–3141 tokens/s and decodes at 35–47 tokens/son one DGX Spark (2k to 32k tokens of context).