Optimizing Chunk Size in Retrieval-Augmented Generation (RAG) Pipelines for Enhanced API Latency

Abstract: Retrieval-Augmented Generation (RAG) has become the standard architecture for integrating large language models (LLMs) with custom knowledge bases. However, in production environments, API latency remains a critical bottleneck. This study investigates the impact of document chunk size on the overall response time of a Naive RAG pipeline. By evaluating three distinct token configurations (250, 500, and 1000) using FAISS and Google's Gemini models, we identify an optimal "sweet spot" that balances vector search efficiency with LLM context processing overhead.