The Reranker Fixed the Set It Was Built On and Nothing Else: A Post Hoc Evaluation of a Standard-Library RAG Harness on Synthetic Sets and 700 Real Questions, with Span Extraction as the Only Significant Gain

Can a small, transparent, standard-library harness measure retrieval-augmented generation (RAG) quality honestly, and does a targeted fix for one failure class hold up outside the data it was built on? This paper reports an exploratory study, and says so in its title: the project was built without a locked plan, and its three hypotheses were written on 2026-11-07, after every result below already existed. They are scored post hoc and carry no confirmatory weight. The harness is a BM25 retriever over sentences with an extractive answer step, three lexical metrics with fixed pass lines, a budgeted judge layer, and a CPU embedding judge, run on synthetic sets of 8, 20, and 32 cases and on deterministic prefix samples of SQuAD validation (500 cases) and HotpotQA distractor (200 cases). H1 (the stemmed-overlap reranker raises the synthetic Deep pass rate) is supported in-distribution only: the Deep set went from 23 to 32 of 32 and the target failure class from 5 to 13 of 13, but 12 of the 13 cases in that class were written after the failure was seen. H2 (the reranker raises gold-in-top-passage on real data) is not supported: on SQuAD the gold answer reached the top passage in 315 of 500 cases without the reranker (Wilson 95 percent interval 0.587 to 0.671) and 319 with it (0.595 to 0.679), 26 helped against 22 hurt, exact sign test p 0.6655; on HotpotQA, 39 to 47 of 200, 14 helped against 6 hurt, p 0.1153. None of the 12 rerank tests survives a Bonferroni correction. H3 (span answers beat sentence answers on lexical pass) is supported, and it is the study's largest measured effect: SQuAD lexical passes rose from 24 to 113 of 500 without rerank (exact McNemar 21 against 110, p below 0.0001) and HotpotQA from 1 to 7 of 200 (0 against 6, p 0.0312). The embedding judge agrees with a gold-derived proxy label at Cohen's kappa 0.5607 for SQuAD sentence answers and 0.7111 for span answers at the shipped thresholds; the judge has not been validated against human labels, and no human labels exist in this study. Every scored result regenerates byte-identically from the committed evidence by one command. The contribution is a negative transfer result, published where it fell, and a small harness whose numbers a reader can check by hand.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24329009
Primary Topic
Information Retrieval and Search Behavior
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

The Reranker Fixed the Set It Was Built On and Nothing Else: A Post Hoc Evaluation of a Standard-Library RAG Harness on Synthetic Sets and 700 Real Questions, with Span Extraction as the Only Significant Gain

Jesun Ahmad Ushno
3 links
Zenodo (CERN European Organization for Nuclear Research)
Information Retrieval and Search Behavior
preprint

The Reranker Fixed the Set It Was Built On and Nothing Else: A Post Hoc Evaluation of a Standard-Library RAG Harness on Synthetic Sets and 700 Real Questions, with Span Extraction as the Only Significant Gain

Jesun Ahmad Ushno
preprint en

Abstract

Can a small, transparent, standard-library harness measure retrieval-augmented generation (RAG) quality honestly, and does a targeted fix for one failure class hold up outside the data it was built on? This paper reports an exploratory study, and says so in its title: the project was built without a locked plan, and its three hypotheses were written on 2026-11-07, after every result below already existed. They are scored post hoc and carry no confirmatory weight. The harness is a BM25 retriever over sentences with an extractive answer step, three lexical metrics with fixed pass lines, a budgeted judge layer, and a CPU embedding judge, run on synthetic sets of 8, 20, and 32 cases and on deterministic prefix samples of SQuAD validation (500 cases) and HotpotQA distractor (200 cases). H1 (the stemmed-overlap reranker raises the synthetic Deep pass rate) is supported in-distribution only: the Deep set went from 23 to 32 of 32 and the target failure class from 5 to 13 of 13, but 12 of the 13 cases in that class were written after the failure was seen. H2 (the reranker raises gold-in-top-passage on real data) is not supported: on SQuAD the gold answer reached the top passage in 315 of 500 cases without the reranker (Wilson 95 percent interval 0.587 to 0.671) and 319 with it (0.595 to 0.679), 26 helped against 22 hurt, exact sign test p 0.6655; on HotpotQA, 39 to 47 of 200, 14 helped against 6 hurt, p 0.1153. None of the 12 rerank tests survives a Bonferroni correction. H3 (span answers beat sentence answers on lexical pass) is supported, and it is the study's largest measured effect: SQuAD lexical passes rose from 24 to 113 of 500 without rerank (exact McNemar 21 against 110, p below 0.0001) and HotpotQA from 1 to 7 of 200 (0 against 6, p 0.0312). The embedding judge agrees with a gold-derived proxy label at Cohen's kappa 0.5607 for SQuAD sentence answers and 0.7111 for span answers at the shipped thresholds; the judge has not been validated against human labels, and no human labels exist in this study. Every scored result regenerates byte-identically from the committed evidence by one command. The contribution is a negative transfer result, published where it fell, and a small harness whose numbers a reader can check by hand.

Zenodo (CERN European Organization for Nuclear Research)
Information Retrieval and Search Behavior
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon