KV Cache Compression Based on Token-Level Redundancy Elimination and Bit-Level Encoding

Large language models (LLMs) face storage and access limitations during development, requiring extensive reads of key-value (KV) cache data for each token generation. This paper designs a hardware-friendly lossless compression and decompression method for KV cache based on Token-level Redundancy Elimination and Bit-level Encoding (TREBE), reducing the amount of KV cache data that needs to be stored. In detail, token-level redundancy is first eliminated by performing subtraction between the same layers of different tokens, followed by bit-level encoding using XOR and bit-plane techniques. TREBE achieves an average compression ratio of 2.1x for the KV cache of LLaMA2-7B and OPT-6.7B models on the WikiText2 and C4 datasets, representing a 1.9x improvement over Huffman coding. With the support of TSMC 28nm process, TREBE operates at 0.9V and 0.8GHz, achieving a throughput of 6.38GB/s. Compared to Huffman coding, energy efficiency is improved by 10x, and area efficiency is enhanced by 31.2x.