Evidence Fragmentation Is a Distinct Failure Mode in Chunked Retrieval: A Controlled Study Across Three Corpora and Seven Encoders

Retrieval-augmented generation (RAG) systems divide documents into fixed-size chunks before indexing, a decision that the literature largely treats as a preprocessing convenience. This study shows that the decision governs a distinct failure mode, referred to here as evidence fragmentation. Using a within-question manipulation that holds the question, its gold evidence, and the document’s token count fixed while varying only the arrangement of that evidence, we find that, in this controlled setting, dispersing evidence across a document causally reduces complete-evidence retrieval by up to 0.471, against a measured noise floor that never exceeds 0.0051. The effect is established on 6200 questions from three corpora spanning scientific and legal text (QASPER, ContractNLI, and CUAD), with two retrieval families and a panel of seven encoders covering five pretraining lineages and a fifteen-fold range of parameter counts; all 252 non-degenerate conditions of the panel are significant, and increasing encoder size does not reduce the penalty. The cause is passage aggregation rather than document structure: the penalty grows monotonically with chunk size, vanishes when a chunk holds a single passage, and peaks at an intermediate context budget. We further show that the natural observational design for this question yields the opposite conclusion, because evidence dispersion is confounded with evidence quantity. Indexing at passage granularity removes the effect at a measured cost of 8–16× more index entries. A larger residual loss remains, and of nine candidate remedies evaluated on held-out questions, five fail, including a cross-encoder reranker that is significantly harmful on one corpus. What distinguishes the remedies that help is that they evaluate a passage together with its neighbors rather than independently. The most effective remedy is also the simplest: delivering contiguous spans of passages ranked by evidence per token, with a mild coupling between neighbors. It improves complete-evidence retrieval by up to 0.185 at the tightest budgets of QASPER and ContractNLI, is significant under every encoder tested on those corpora, and is not improved upon by more elaborate schemes built on the same diagnosis. On CUAD, where evidence passages are long relative to the budget, it falls below the flat baseline, and a multi-representation scheme is the safer choice. The findings indicate that chunk size is a retrieval-behavior decision and that the delivery unit should be chosen per query rather than fixed at indexing time.