When Document Metrics Hide Passage Damage: Auditing Detection and Selective Repair in Retrieval-Augmented Generation
Retrieving a relevant document does not ensure that a retrieval-augmented generation system supplies the passages needed to answer a question. We investigate a narrower operational problem: when can document-level evaluation miss degradation in a passage ranking, and can a bounded semantic check guide a useful repair? We formalize a slot-preserving within-document intervention and evaluate a frozen audit-and-repair procedure on 424 QASPER questions from 250 reserved source papers, following separate fitting and calibration. Reversing passages within their existing document slots leaves every document score unchanged while decreasing mean passage nDCG@10 by 0.208. A sampled checker adds 0.122 harmful-case recall beyond an integrity guard. Selective repair improves passage nDCG by 0.0134 over the guard in a fault-heavy challenge, but misses the prespecified 0.030 practical target, fails the unmodified-input damage bound, and performs worse than the guard in a 10% fault scenario. Full-pool reranking with the smaller checker has better mean quality at lower incremental GPU cost in the challenge, further limiting the proposed policy's value. A separate frozen answer experiment covers 100 source papers and 700 conditions: selective repair raises Answer F1 after within-document reversal from 0.155 to 0.189, below the unmodified 0.242. Adapted eRAG and RAGChecker diagnostics associate with answer quality, but do not establish reliable prediction of repair benefit. The checker detected damage that document scores missed, but routing on its alerts did not yield a consistent repair benefit. Independent human assessment of the answers remains pending.
Authors
- S M Abdulla Al Mamun
Institutions
- University of Manchester (GB)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24368226
- Primary Topic
- Topic Modeling
- Type
- preprint