KV Cache Group Quantization Based on vLLM Inference Engine

Inference with Large Language Models (LLMs) is a hot topic in the field of Artificial Intelligence. vLLM is an efficient inference engine for LLMs, which solves the memory fragmentation of KV Cache by chunking it and is widely used in large language model service systems by virtue of its efficient memory management mechanism and excellent inference performance. Quantization greatly reduces the memory occupation of parameters by approximating the representation of parameters from high bit-width data to low bit-width data. Quantization is a key technique to accelerate model inference. However, the vLLM inference engine only supports quantization of models, and in addition to the models, the KV Cache in the inference process also occupies a large amount of memory. To address the above problems, we propose a KV Cache group quantization method based on the vLLM inference engine, which reduces the memory consumption in the model inference process by quantizing the KV Cache and optimizes the computation introduced by quantization and dequantization at the same time. We evaluate our method on LLaMA2-7B, LLaMA3-8B, and DeepSeek-R1-Distill-LLaMA3-8B models. Experimental results demonstrate that under concurrent multi-request scenarios, our approach improves inference throughput by up to 18% while maintaining low accuracy loss.