MiniKV: Pushing the Limits of 2-Bit KV Cache via Compression and System Co-Design for Efficient Long Context Inference

State-of-the-art 2-bit KV cache quantization techniques achieve excellent results in accelerating LLM inference while retaining accuracy on long context tasks.However, further pushing the compression ratio fails to deliver performance gains.In this work, we revisit these approaches by considering, additionally, adaptive KV methods that retain LLM accuracy with only a subset of KV states.This leads us to propose a method based on 2-bit KV cache quantization with adaptive KV policies.In addition, we take an algorithm and system co-design approach by developing hardwarefriendly kernels to accelerate LLM inference while making MiniKV compatible with existing memory-efficient attention techniques such as FlashAttention, effectively translating algorithmic improvements into system performance gains.Experiments on a wide range of long context tasks show that MiniKV effectively achieves >80% KV cache compression while retaining accuracy, outperforming state-of-theart methods while achieving excellent latency, throughput, and memory consumption improvements in long context inference.