Hierarchical KV Cache Clustering for Memory-Efficient Transformer Inference

The paper presents a hierarchical clustering framework for online Key-Value (KV) cache compression in Transformer-based large language models. The proposed approach organizes structurally similar KV representations into compact prototype clusters to reduce memory consumption while preserving contextual information during long-context autoregressive inference. The manuscript introduces a localized similarity-driven clustering strategy, Progressive Similarity Association (PSA), together with Adaptive Regional Partitioning (ARP) and Prototype Aggregation (PAG) to enable efficient online cache compression with low computational overhead. The paper also includes theoretical analysis, empirical observations, experimental evaluation, and ablation studies across long-context language modeling benchmarks. This deposit serves as an open-access research manuscript intended to facilitate discussion, reproducibility, and future research in efficient Transformer inference and long-context large language models. Keywords: Large Language Models, Transformer Inference, KV Cache Compression, Long-Context Modeling, Efficient AI, Memory Optimization, Hierarchical Clustering, Attention Mechanisms, Autoregressive Inference.