MEMORY AND LATENCY OPTIMIZATION IN LARGE LANGUAGE MODELS VIA KEY-VALUE CACHING

The article examines the problem of optimizing memory and latency during the inference of large language models (LLMs) by efficiently managing the key–value cache, a central element of the autoregressive text generation mechanism. The relevance of the topic is driven by the exponential growth in the scale of modern LLMs and the sharp increase in computational and monetary costs in the inference phase, where memory capacity and bandwidth become the limiting factors of performance. The study aims to systematize and comparatively analyze contemporary approaches to KV-cache optimization at three levels, system, kernel, and algorithmic, while identifying their synergistic effects and engineering trade-offs. The novelty of the work lies in proposing a unified taxonomic structure that brings together heterogeneous methods traditionally considered in isolation: memory virtualization mechanisms (PagedAttention), IO-aware computational kernels (FlashAttention), cache compression methods via quantization (FP8, INT8), and intelligent token eviction policies (AhaKV, SAGE-KV, Attention-Gate). The study demonstrates that maximum efficiency is achieved by combining system and algorithmic techniques. Page-based memory organization eliminates fragmentation and increases GPU utilization, while IO-optimized attention kernels reduce latency. Additionally, quantization, combined with adaptive token eviction, ensures the scaling of context length while preserving generation quality. It is substantiated that further advances hinge on the creation of dynamic, self-learning policies for KV-cache management and on GPU architectures specialized for the memory access patterns of LLMs, as well as on integrating privacy-aware mechanisms that prevent data leakage when caches are shared among users. The article will be helpful to researchers and engineers involved in designing LLM inference systems, optimizing GPU performance, and developing highly efficient AI architectures.