When Document Metrics Hide Passage Damage: Auditing Detection and Selective Repair in Retrieval-Augmented Generation

Retrieving a relevant document does not ensure that a retrieval-augmented generation system supplies the passages needed to answer a question. We investigate a narrower operational problem: when can document-level evaluation miss degradation in a passage ranking, and can a bounded semantic check guide a useful repair? We formalize a slot-preserving within-document intervention and evaluate a frozen audit-and-repair procedure on 424 QASPER questions from 250 reserved source papers, following separate fitting and calibration. Reversing passages within their existing document slots leaves every document score unchanged while decreasing mean passage nDCG@10 by 0.208. A sampled checker adds 0.122 harmful-case recall beyond an integrity guard. Selective repair improves passage nDCG by 0.0134 over the guard in a fault-heavy challenge, but misses the prespecified 0.030 practical target, fails the unmodified-input damage bound, and performs worse than the guard in a 10% fault scenario. Full-pool reranking with the smaller checker has better mean quality at lower incremental GPU cost in the challenge, further limiting the proposed policy's value. A separate frozen answer experiment covers 100 source papers and 700 conditions: selective repair raises Answer F1 after within-document reversal from 0.155 to 0.189, below the unmodified 0.242. Adapted eRAG and RAGChecker diagnostics associate with answer quality, but do not establish reliable prediction of repair benefit. The checker detected damage that document scores missed, but routing on its alerts did not yield a consistent repair benefit. Independent human assessment of the answers remains pending.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24368226
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

When Document Metrics Hide Passage Damage: Auditing Detection and Selective Repair in Retrieval-Augmented Generation

S M Abdulla Al Mamun
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

When Document Metrics Hide Passage Damage: Auditing Detection and Selective Repair in Retrieval-Augmented Generation

S M Abdulla Al Mamun
preprint en

Abstract

Retrieving a relevant document does not ensure that a retrieval-augmented generation system supplies the passages needed to answer a question. We investigate a narrower operational problem: when can document-level evaluation miss degradation in a passage ranking, and can a bounded semantic check guide a useful repair? We formalize a slot-preserving within-document intervention and evaluate a frozen audit-and-repair procedure on 424 QASPER questions from 250 reserved source papers, following separate fitting and calibration. Reversing passages within their existing document slots leaves every document score unchanged while decreasing mean passage nDCG@10 by 0.208. A sampled checker adds 0.122 harmful-case recall beyond an integrity guard. Selective repair improves passage nDCG by 0.0134 over the guard in a fault-heavy challenge, but misses the prespecified 0.030 practical target, fails the unmodified-input damage bound, and performs worse than the guard in a 10% fault scenario. Full-pool reranking with the smaller checker has better mean quality at lower incremental GPU cost in the challenge, further limiting the proposed policy's value. A separate frozen answer experiment covers 100 source papers and 700 conditions: selective repair raises Answer F1 after within-document reversal from 0.155 to 0.189, below the unmodified 0.242. Adapted eRAG and RAGChecker diagnostics associate with answer quality, but do not establish reliable prediction of repair benefit. The checker detected damage that document scores missed, but routing on its alerts did not yield a consistent repair benefit. Independent human assessment of the answers remains pending.

Zenodo (CERN European Organization for Nuclear Research)
University of Manchester (GB)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon