VANTAGE-V-AI: Bounded-Memory KV-Cache Compression and Streaming Inference Runtime for Large Language Models

Autoregressive generation in Large Language Models (LLMs) deployed within extended multi-turn dialogs and autonomous agentic workflows suffers from an acute memory wall. Because conventional Key-Value (KV) caching retains representations for all preceding tokens, memory consumption scales linearly O(N) with sequence length and batch size, precipitating catastrophic Out-Of-Memory (OOM) failures or massive resource underutilization. In this work, we present VANTAGE-V-AI (v8.2.0), a high-performance inference runtime designed to bound KV-cache memory consumption to a strictly fixed budget O(W) while maintaining numerical parity and stability. VANTAGE integrates three complementary mechanisms: (1) Streaming Attention Sinks, which persistently retain initial anchor tokens to anchor attention distribution scores alongside a recency sliding window, eliminating softmax entropy collapse; (2) 2-Bit Packed Ternary Quantization ({-1, 0, +1}) packing four ternary values per byte, achieving an 8x reduction in KV-cache storage; and (3) a drop-in runtime architecture that wraps standard Hugging Face Transformer decoders in two lines of code without retraining. We empirically benchmark VANTAGE on dual NVIDIA Tesla T4 GPUs across a 1,024-token prompt using Qwen/Qwen2.5-3B-Instruct. Following MLSys asynchronous execution protocols, we report end-to-end wall-clock timing including host-device PCIe transfers and multi-GPU synchronization. The system establishes 100.0% pairwise token parity (64/64 tokens) against the uncompressed baseline, achieves 0.988 average coherence in long-context Needle-In-A-Haystack retrieval, and prevents explosive softmax perplexity divergence.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-15
DOI
https://doi.org/10.5281/zenodo.24243286
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

VANTAGE-V-AI: Bounded-Memory KV-Cache Compression and Streaming Inference Runtime for Large Language Models

Vo Anh Khoa Nguyen
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
article

VANTAGE-V-AI: Bounded-Memory KV-Cache Compression and Streaming Inference Runtime for Large Language Models

Vo Anh Khoa Nguyen
article en

Abstract

Autoregressive generation in Large Language Models (LLMs) deployed within extended multi-turn dialogs and autonomous agentic workflows suffers from an acute memory wall. Because conventional Key-Value (KV) caching retains representations for all preceding tokens, memory consumption scales linearly O(N) with sequence length and batch size, precipitating catastrophic Out-Of-Memory (OOM) failures or massive resource underutilization. In this work, we present VANTAGE-V-AI (v8.2.0), a high-performance inference runtime designed to bound KV-cache memory consumption to a strictly fixed budget O(W) while maintaining numerical parity and stability. VANTAGE integrates three complementary mechanisms: (1) Streaming Attention Sinks, which persistently retain initial anchor tokens to anchor attention distribution scores alongside a recency sliding window, eliminating softmax entropy collapse; (2) 2-Bit Packed Ternary Quantization ({-1, 0, +1}) packing four ternary values per byte, achieving an 8x reduction in KV-cache storage; and (3) a drop-in runtime architecture that wraps standard Hugging Face Transformer decoders in two lines of code without retraining. We empirically benchmark VANTAGE on dual NVIDIA Tesla T4 GPUs across a 1,024-token prompt using Qwen/Qwen2.5-3B-Instruct. Following MLSys asynchronous execution protocols, we report end-to-end wall-clock timing including host-device PCIe transfers and multi-GPU synchronization. The system establishes 100.0% pairwise token parity (64/64 tokens) against the uncompressed baseline, achieves 0.988 average coherence in long-context Needle-In-A-Haystack retrieval, and prevents explosive softmax perplexity divergence.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon