RTT- or Bandwidth-Bound? Demystifying the KV Cache Transfer in Large Language Model Serving
Modern large language model (LLM) serving systems increasingly adopt a prefill-decode disaggregation architecture to enhance inference efficiency. While this design improves resource utilization, it introduces latency due to the transfer of key-value (KV) cache. The community has generally assumed that this latency is bandwidth-bound and can be effectively mitigated by high-speed interconnects.
Authors
- Mowei Wang (ORCID: https://orcid.org/0000-0001-9085-2247)
- Zihan Jiang (ORCID: https://orcid.org/0000-0003-0632-7402)
- Z. Zhang
- Shengnan Yue
- Yu Yan
- Weiqiang Cheng (ORCID: https://orcid.org/0000-0002-4459-8847)
Institutions
- China Mobile (China) (CN)
- University of Mobile (US)
- Huawei Technologies (China) (CN)
- Huawei Technologies (United States) (US)
Publication Details
- Published
- 2025-09-02
- DOI
- https://doi.org/10.1145/3748273.3749196
- Primary Topic
- Topic Modeling
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Key Research and Development Program of China