Exact Evaluation of Adaptive Prefix-Cache Continuations
In LLM serving with prefix caching, several request orders can incur the same immediate prefill work yet leave different cache states. We give an exact method to score a known, finite continuation from each state under least-recently-used eviction and entry-time charging of whole prefix paths. The continuation can select its next request using current hits. Equivalent starting states can share scores, and a sufficient persistence test can determine a score without replay. In one previously examined Mooncake tool/agent condition (128 blocks, decision output), the two shortcuts reduced full replays from 267,672 to 46,610. The complete-state and focused-state implementations used the same shortcuts and agreed on replay counts in each of eight conditions, as expected. The complete-state implementation used more mean native CPU in six of the eight. In the highlighted condition the focused-state implementation took 66.59 microseconds per decision versus 89.00 for the complete-state implementation. Local caller wall time fell 16.9%. The policy saved 0.204% of modeled prefill work at 128 blocks and 1.001% at 32 on Mooncake, and zero on two external episodes. We have not established an end-to-end benefit for an independent planner.
Authors
- Mohammad Ali Seyed Mahmoud
Institutions
- Georgia Institute of Technology (US)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.5281/zenodo.24177974
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- preprint