Why Did the Prompt Cache Miss? Per-Request Cause Attribution for Agent Workloads

Agents resend the whole conversation on every model call. Prompt caching makes the repeated part cheap, so one changed byte near the top of the prompt costs the whole prompt again. Providers and inference engines report how many tokens missed the cache, not why. We present Prefixwise, a tool that names the cause of each cache miss, per request, on hosted APIs (Anthropic, OpenAI and compatible) and on self-hosted engines (vLLM, SGLang). It splits each request into blocks in the order the provider renders them and keeps three fingerprints per block (raw, canonical, masked), so a reordered tool or a ticking timestamp is told apart from a real edit. The first changed block names the cause, and the verdict is checked against the provider’s own usage report. Behind an engine it joins the engine’s KV-cache events to the request, which turns “nothing was read” into a confirmed eviction, a routing miss across replicas, or a hit restored from a CPU tier. Each miss is priced, and two fixes are applied live. On 1,772 changes injected into real Claude Code trajectories and generated conversations on vLLM and SGLang, every change was seen, 1,746 were named as the primary cause and 26 cost nothing, none was named as another cause, and there were no false alarms on 3,622 append-only turns. It agreed with Anthropic’s own diagnosis on 32 of 32 misses Anthropic named. Evictions and routing misses were confirmed 5 of 5 from engine events. A fix recovered 97.5% to 99.9% of the saving it predicted, with task success unchanged. The proxy adds about 3 ms per request and stores hashes and counts, never prompt text. Prefixwise is open source (MIT): https://github.com/tejachavali/prefixwise.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24331593
Primary Topic
Software System Performance and Reliability
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Why Did the Prompt Cache Miss? Per-Request Cause Attribution for Agent Workloads

Tejaswi Chavali
Zenodo (CERN European Organization for Nuclear Research)
Software System Performance and Reliability
preprint

Why Did the Prompt Cache Miss? Per-Request Cause Attribution for Agent Workloads

Tejaswi Chavali
preprint en

Abstract

Agents resend the whole conversation on every model call. Prompt caching makes the repeated part cheap, so one changed byte near the top of the prompt costs the whole prompt again. Providers and inference engines report how many tokens missed the cache, not why. We present Prefixwise, a tool that names the cause of each cache miss, per request, on hosted APIs (Anthropic, OpenAI and compatible) and on self-hosted engines (vLLM, SGLang). It splits each request into blocks in the order the provider renders them and keeps three fingerprints per block (raw, canonical, masked), so a reordered tool or a ticking timestamp is told apart from a real edit. The first changed block names the cause, and the verdict is checked against the provider’s own usage report. Behind an engine it joins the engine’s KV-cache events to the request, which turns “nothing was read” into a confirmed eviction, a routing miss across replicas, or a hit restored from a CPU tier. Each miss is priced, and two fixes are applied live. On 1,772 changes injected into real Claude Code trajectories and generated conversations on vLLM and SGLang, every change was seen, 1,746 were named as the primary cause and 26 cost nothing, none was named as another cause, and there were no false alarms on 3,622 append-only turns. It agreed with Anthropic’s own diagnosis on 32 of 32 misses Anthropic named. Evictions and routing misses were confirmed 5 of 5 from engine events. A fix recovered 97.5% to 99.9% of the saving it predicted, with task success unchanged. The proxy adds about 3 ms per request and stores hashes and counts, never prompt text. Prefixwise is open source (MIT): https://github.com/tejachavali/prefixwise.

Zenodo (CERN European Organization for Nuclear Research)
Software System Performance and Reliability
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon