Do LLMs Know When They Are Wrong? Resampling vs. Paraphrase Disagreement as Black-Box Error Signals

Disagreement across paraphrases flagged wrong answers better than disagreement across resamples at the same call budget, but both signals are weak because most errors are confident. On DeepSeek-chat with 1,600 MMLU and 1,600 TriviaQA questions, AUROC for detecting errors with 3 extra answers was 0.695 (paraphrase) vs 0.646 (resample) on MMLU (difference +0.049, 95% CI 0.011 to 0.089) and 0.785 vs 0.668 on TriviaQA (+0.117, CI 0.065 to 0.169). Both differences remain significant after Holm correction. About 70% of errors are confident (at least 4 of 5 samples agree on the same wrong answer). Resampling flags 10% of them and paraphrasing 40%, at a higher false-alarm rate (9.4% vs 4.0%). Caveats: one model, DeepSeek writes and checks its own paraphrases, the MMLU gap shrinks to +0.031 (CI -0.009 to 0.071) when questions with missing paraphrases are kept, and 27% to 30% of audited "errors" were label noise or ambiguous questions. These are preliminary results. Preliminary results. Single model (DeepSeek-chat), 1,600 MMLU and 1,600 TriviaQA questions. Hand audits were first drafted with an LLM (Claude) and entered by the author; the report text was written with LLM assistance.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24289691
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Do LLMs Know When They Are Wrong? Resampling vs. Paraphrase Disagreement as Black-Box Error Signals

Prerna Sachdeva
1 links
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Do LLMs Know When They Are Wrong? Resampling vs. Paraphrase Disagreement as Black-Box Error Signals

Prerna Sachdeva
preprint en

Abstract

Disagreement across paraphrases flagged wrong answers better than disagreement across resamples at the same call budget, but both signals are weak because most errors are confident. On DeepSeek-chat with 1,600 MMLU and 1,600 TriviaQA questions, AUROC for detecting errors with 3 extra answers was 0.695 (paraphrase) vs 0.646 (resample) on MMLU (difference +0.049, 95% CI 0.011 to 0.089) and 0.785 vs 0.668 on TriviaQA (+0.117, CI 0.065 to 0.169). Both differences remain significant after Holm correction. About 70% of errors are confident (at least 4 of 5 samples agree on the same wrong answer). Resampling flags 10% of them and paraphrasing 40%, at a higher false-alarm rate (9.4% vs 4.0%). Caveats: one model, DeepSeek writes and checks its own paraphrases, the MMLU gap shrinks to +0.031 (CI -0.009 to 0.071) when questions with missing paraphrases are kept, and 27% to 30% of audited "errors" were label noise or ambiguous questions. These are preliminary results. Preliminary results. Single model (DeepSeek-chat), 1,600 MMLU and 1,600 TriviaQA questions. Hand audits were first drafted with an LLM (Claude) and entered by the author; the report text was written with LLM assistance.

Zenodo (CERN European Organization for Nuclear Research)
Stony Brook University (US)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon