CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion
Large language models (LLMs) often incorporate multiple text chunks in their inputs to provide the necessary contexts. To speed up the prefill of the long LLM inputs, one can pre-compute the KV cache of a text and re-use the KV cache when the context is reused as the prefix of another LLM input. However, the reused text chunks are not always the input prefix, which makes precomputed KV caches not directly usable since they ignore the text's cross-attention with the preceding texts. Thus, the benefits of reusing KV caches remain largely unrealized.
Authors
- Hanchen Li (ORCID: https://orcid.org/0009-0005-9980-028X)
- Yihua Cheng (ORCID: https://orcid.org/0009-0006-3924-6886)
- Siddhant Ray (ORCID: https://orcid.org/0000-0003-0265-2144)
- Kuntai Du (ORCID: https://orcid.org/0000-0002-3964-4079)
- Jiayi Yao (ORCID: https://orcid.org/0000-0002-8588-4356)
- Shan Lu (ORCID: https://orcid.org/0000-0002-0757-4600)
- Yuhan Liu (ORCID: https://orcid.org/0009-0002-5957-5071)
- Qizheng Zhang (ORCID: https://orcid.org/0009-0009-3208-4601)
- Junchen Jiang (ORCID: https://orcid.org/0000-0002-6877-1683)
Institutions
- University of Chicago (US)
- Microsoft Research (United Kingdom) (GB)
- University of Hong Kong (HK)
- Stanford University (US)
Publication Details
- Published
- 2025-03-26
- DOI
- https://doi.org/10.1145/3689031.3696098
- Citations
- 26
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 64.51