LateSplit: Lightweight Post-Retrieval Chunking for Query-Aligned Text Segmentation in RAG Systems

Retrieval-Augmented Generation (RAG) has become indispensable for knowledge-intensive applications, enhancing large language models (LLMs) by integrating external information. However, the efficacy of RAG systems critically depends on text chunking strategies, where conventional pre-retrieval methods often produce misaligned or fragmented content, impairing downstream tasks. This paper introduces LateSplit, a hybrid chunking framework that combines conventional pre- retrieval chunking with novel post-retrieval refinement to better align documents with query intent. Building on standard chunking approaches, LateSplit performs dynamic boundary adjustment after retrieval, preserving logical coherence while improving relevance. As a lightweight, plug-and-play solution, LateSplit reduces computational overhead by 1.9x compared to semantic chunking techniques while achieving precision improvements of up to 44.2 % and IoU gains of 20.4 % across diverse datasets. Experimental results demonstrate its flexibility and practicality, offering a cost-effective enhancement for RAG pipelines.