Applying Certified L∞-Bounded Compression to LLM Serving: Portable Saved State, Shared Pools and Mixed-Generation GPUs
Earlier work (Parts~I and~II of the author's series on certified $L_\infty$-bounded compression) compresses a language model's weights and key--value (KV) cache under a certified $L_\infty$ contract: every stored number is within a stated tolerance of the original. This paper asks what the contract makes possible in a \emph{serving} system. A saved prefix (the KV cache and recurrent state of the beginning of a conversation) is then \emph{data defined by its numbers}, not by the GPU that made it. We use this in four ways: a transfer signature and per-piece hashes that decide whether a saved prefix may be used; a copy-or-recompute rule priced from a measured link speed; a host-RAM pool shared by the servers of a machine and a RAM-only machine that holds copies and overflow; and layer splits over cards and machines. On a DGX Spark (Blackwell) and a V100 (Volta), a prefix of 12\,288 tokens (496~MB) saved on one was restored on the other, giving the same greedy answer in 10.3~s against 40.2~s for computing it cold. A 27-billion-parameter model whose pack is 13.6~GB runs on one 16~GB V100 with host-RAM offload. All numbers are single runs on three machines and a Wi-Fi link; the large-server layouts we draw are simulations, not results.
Authors
- C.N. Sam (ORCID: https://orcid.org/0000-0001-8267-4815)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-16
- DOI
- https://doi.org/10.5281/zenodo.24267207
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- article
- Field-Weighted Citation Impact
- 0.00