Why Did the Prompt Cache Miss? Per-Request Cause Attribution for Agent Workloads
Agents resend the whole conversation on every model call. Prompt caching makes the repeated part cheap, so one changed byte near the top of the prompt costs the whole prompt again. Providers and inference engines report how many tokens missed the cache, not why. We present Prefixwise, a tool that names the cause of each cache miss, per request, on hosted APIs (Anthropic, OpenAI and compatible) and on self-hosted engines (vLLM, SGLang). It splits each request into blocks in the order the provider renders them and keeps three fingerprints per block (raw, canonical, masked), so a reordered tool or a ticking timestamp is told apart from a real edit. The first changed block names the cause, and the verdict is checked against the provider’s own usage report. Behind an engine it joins the engine’s KV-cache events to the request, which turns “nothing was read” into a confirmed eviction, a routing miss across replicas, or a hit restored from a CPU tier. Each miss is priced, and two fixes are applied live. On 1,772 changes injected into real Claude Code trajectories and generated conversations on vLLM and SGLang, every change was seen, 1,746 were named as the primary cause and 26 cost nothing, none was named as another cause, and there were no false alarms on 3,622 append-only turns. It agreed with Anthropic’s own diagnosis on 32 of 32 misses Anthropic named. Evictions and routing misses were confirmed 5 of 5 from engine events. A fix recovered 97.5% to 99.9% of the saving it predicted, with task success unchanged. The proxy adds about 3 ms per request and stores hashes and counts, never prompt text. Prefixwise is open source (MIT): https://github.com/tejachavali/prefixwise.
Authors
- Tejaswi Chavali
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-17
- DOI
- https://doi.org/10.5281/zenodo.24331593
- Primary Topic
- Software System Performance and Reliability
- Type
- preprint