Efficient pipeline for retrieval-augmented generation system under big data

Retrieval-augmented generation (RAG) is a methodology that integrates external knowledge through knowledge retrieval techniques such as approximate nearest neighbor search (ANNS), significantly enhancing the generation quality of large language models (LLMs). However, as external knowledge bases continue to expand, the storage requirements for ANNS indexes also surge, making the storage of massive data in memory impractical. This has further promoted the development and application of disk-based ANNS, but it significantly increases the response time of RAG systems. To address this issue, this paper proposes the PipeRAG method, which effectively overlaps the latency of knowledge retrieval and LLM inference by pipelining the disk-based ANNS and LLM prefill processes, thereby enhancing the overall performance of RAG systems while ensuring retrieval accuracy. Specifically, PipeRAG features two core designs: the “ANNS adaptive prefetching mechanism" and the “RAG dynamic pipeline scheduling strategy". The former design adjusts the prefetching speed in real time based on the current retrieval status, finding an optimal balance between performance and accuracy. The latter design dynamically adjusts the size of prefilling tasks by considering both ANNS prefetching speed and LLM chunked-prefill latency to achieve optimal pipeline efficiency. Our evaluations under real-world production workloads show that PipeRAG successfully reduces the response latency of RAG systems using disk-based ANNS by 25% to 71%, while maintaining extremely low recall loss.