Razor: Learning What to Forget in the KV Cache with a Description-Length Price

The key-value (KV) cache of a transformer grows linearly with context length, and at long context it takes more GPU memory than the model weights. We present RAZOR, a method for training a small gate that decides, when each token is written to the cache, whether to keep it. The base model stays frozen. The gate is trained to leave the model's output distribution unchanged, measured as KL divergence to the uncompressed model, while paying a price for every token it keeps. The price is updated by dual ascent until a target memory budget is met, so no penalty weight has to be tuned. Keep/drop decisions are binary during training, with straight-through gradients, and the gate therefore trains on the same cache it produces at inference. Trained for 39 minutes on one T4 GPU with documents of at most 1k tokens, the gate (1.7% of model parameters) lets Qwen3-1.7B evict 76% of its KV cache on RULER-4k while scoring 86.6 +/- 0.6 over five seeds, against 88.9 without compression and 86.5 for KVzip+, which requires a second pass over the context. With architecture and data held fixed, it outperforms a gate trained on KVzap's regression targets by 2.3-7.1 points. The same gate transfers to 16k-token RULER and 100k-token InfiniteBench, and lets Qwen3-4B read 123k tokens on a 16 GB GPU, where the uncompressed model runs out of memory at 61k; at 29k tokens, decoding is 2.9x faster. Finally, across four budgets the learned price matches the slope of the cache's rate-distortion curve, which gives a direct measure of what a unit of KV memory is worth to the model.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24183935
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Razor: Learning What to Forget in the KV Cache with a Description-Length Price

Himansu Rawal
1 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Razor: Learning What to Forget in the KV Cache with a Description-Length Price

Himansu Rawal
preprint en

Abstract

The key-value (KV) cache of a transformer grows linearly with context length, and at long context it takes more GPU memory than the model weights. We present RAZOR, a method for training a small gate that decides, when each token is written to the cache, whether to keep it. The base model stays frozen. The gate is trained to leave the model's output distribution unchanged, measured as KL divergence to the uncompressed model, while paying a price for every token it keeps. The price is updated by dual ascent until a target memory budget is met, so no penalty weight has to be tuned. Keep/drop decisions are binary during training, with straight-through gradients, and the gate therefore trains on the same cache it produces at inference. Trained for 39 minutes on one T4 GPU with documents of at most 1k tokens, the gate (1.7% of model parameters) lets Qwen3-1.7B evict 76% of its KV cache on RULER-4k while scoring 86.6 +/- 0.6 over five seeds, against 88.9 without compression and 86.5 for KVzip+, which requires a second pass over the context. With architecture and data held fixed, it outperforms a gate trained on KVzap's regression targets by 2.3-7.1 points. The same gate transfers to 16k-token RULER and 100k-token InfiniteBench, and lets Qwen3-4B read 123k tokens on a 16 GB GPU, where the uncompressed model runs out of memory at 61k; at 29k tokens, decoding is 2.9x faster. Finally, across four budgets the learned price matches the slope of the cache's rate-distortion curve, which gives a direct measure of what a unit of KV memory is worth to the model.

Zenodo (CERN European Organization for Nuclear Research)
Kyrgyz-Türkish Manas Üniversity (KG)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon