Right Document, Wrong Sentence: A Pre-registered Abstention Study of a Local Multi-Document Research Assistant, in Which a Frozen Threshold Refused 90 Percent of Unanswerable Questions and the Extracted Sentence Held the Answer Only 41 Percent of the

A retrieval tool that answers questions from a local document folder faces two distinct failure modes: it can answer questions the documents cannot support, and it can point at the right document while returning the wrong sentence. This study measures both, under rules frozen in advance. The tool under test is a local research assistant built by the author: it chunks a folder of documents into 120-word windows, embeds them locally with minishlab/potion-base-8M through model2vec, retrieves the single top chunk by cosine similarity, returns the sentence inside that chunk most similar to the question with a citation, and abstains when the top chunk scores below a frozen threshold of T = 0.40. The corpus (16 self-written technical documents, 5,124 words, 62 chunks), the question battery (30 in-corpus questions with gold documents and gold phrases, 20 out-of-corpus questions including deliberate vocabulary traps), the threshold, and three hypotheses with numeric verdict lines were committed together in a pre-registration (2026-11-15) before the tool's measurement code was written. All three hypotheses were CONFIRMED. H1: 29 of 30 in-corpus questions (0.967) were answered with the gold document cited, against a frozen line of 0.80. H2: 18 of 20 out-of-corpus questions (0.900) were refused, against a line of 0.80 and against the reference failure that motivated the study, an earlier system in the same programme that abstained on only 0.167 of its adversarial questions. H3: median per-question latency was 0.0013 s over 250 measurements, against a line of 2.0 s. The frozen secondary measure exposes the weak stage: of the 29 answered in-corpus questions, only 12 (0.414) returned the sentence containing the gold phrase, so the citation is the reliable product and the extracted sentence is a pointer. The failures are three and all sit in the overlap band between the two score distributions (in-corpus top scores 0.397 to 0.762, out-of-corpus 0.090 to 0.507): one in-corpus abstention at 0.3967, and two vocabulary traps answered at 0.4162 and 0.5071. One correction to the study record is reported in full: the repository's RESULTS.md states that the abstained question's correct chunk was the top retrieval result, but re-execution with the threshold disabled shows the top chunk came from a different document, so the abstention also masked a wrong-document retrieval. The claims are deliberately narrow: one small self-written corpus, one embedding model, one machine, one threshold. Within those bounds, a threshold frozen before the fact produced a tool that refuses most of what it cannot support, where the reference system in the motivating record refused almost nothing, at a cost of one in-corpus refusal, and the remaining defect is sentence selection, not retrieval or refusal.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24328893
Primary Topic
Information Retrieval and Search Behavior
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Right Document, Wrong Sentence: A Pre-registered Abstention Study of a Local Multi-Document Research Assistant, in Which a Frozen Threshold Refused 90 Percent of Unanswerable Questions and the Extracted Sentence Held the Answer Only 41 Percent of the

Jesun Ahmad Ushno
6 links
Zenodo (CERN European Organization for Nuclear Research)
Information Retrieval and Search Behavior
preprint

Right Document, Wrong Sentence: A Pre-registered Abstention Study of a Local Multi-Document Research Assistant, in Which a Frozen Threshold Refused 90 Percent of Unanswerable Questions and the Extracted Sentence Held the Answer Only 41 Percent of the

Jesun Ahmad Ushno
preprint en

Abstract

A retrieval tool that answers questions from a local document folder faces two distinct failure modes: it can answer questions the documents cannot support, and it can point at the right document while returning the wrong sentence. This study measures both, under rules frozen in advance. The tool under test is a local research assistant built by the author: it chunks a folder of documents into 120-word windows, embeds them locally with minishlab/potion-base-8M through model2vec, retrieves the single top chunk by cosine similarity, returns the sentence inside that chunk most similar to the question with a citation, and abstains when the top chunk scores below a frozen threshold of T = 0.40. The corpus (16 self-written technical documents, 5,124 words, 62 chunks), the question battery (30 in-corpus questions with gold documents and gold phrases, 20 out-of-corpus questions including deliberate vocabulary traps), the threshold, and three hypotheses with numeric verdict lines were committed together in a pre-registration (2026-11-15) before the tool's measurement code was written. All three hypotheses were CONFIRMED. H1: 29 of 30 in-corpus questions (0.967) were answered with the gold document cited, against a frozen line of 0.80. H2: 18 of 20 out-of-corpus questions (0.900) were refused, against a line of 0.80 and against the reference failure that motivated the study, an earlier system in the same programme that abstained on only 0.167 of its adversarial questions. H3: median per-question latency was 0.0013 s over 250 measurements, against a line of 2.0 s. The frozen secondary measure exposes the weak stage: of the 29 answered in-corpus questions, only 12 (0.414) returned the sentence containing the gold phrase, so the citation is the reliable product and the extracted sentence is a pointer. The failures are three and all sit in the overlap band between the two score distributions (in-corpus top scores 0.397 to 0.762, out-of-corpus 0.090 to 0.507): one in-corpus abstention at 0.3967, and two vocabulary traps answered at 0.4162 and 0.5071. One correction to the study record is reported in full: the repository's RESULTS.md states that the abstained question's correct chunk was the top retrieval result, but re-execution with the threshold disabled shows the top chunk came from a different document, so the abstention also masked a wrong-document retrieval. The claims are deliberately narrow: one small self-written corpus, one embedding model, one machine, one threshold. Within those bounds, a threshold frozen before the fact produced a tool that refuses most of what it cannot support, where the reference system in the motivating record refused almost nothing, at a cost of one in-corpus refusal, and the remaining defect is sentence selection, not retrieval or refusal.

Zenodo (CERN European Organization for Nuclear Research)
Information Retrieval and Search Behavior
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon