Answer Length, Not Truth: Re-evaluating White-Box Hallucination Detectors Under Fair Protocols

White-box hallucination detectors read a language model's internal states to flag hallucinated answers, and recent work reports AUROC above 98%. We re-evaluate two such detectors, MultiHaluDet and ICR Probe, on Mistral-7B and Llama-3.1-8B under one controlled protocol: shared folds, paired bootstrap confidence intervals, and pass/fail criteria fixed before each experiment. On HaluEval QA, answer length alone reaches 0.97 AUROC; when only answers of equal length are compared, MultiHaluDet falls from 0.985 to 0.805-0.815, and its six-classifier stacking stage adds nothing over averaging its deep models. When the models answer TriviaQA and Natural Questions themselves, MultiHaluDet scores below trained token log-probabilities in seven of eight comparisons and ties in the eighth, and ICR Probe scores below them in all eight. A logistic-regression probe on the full hidden state of the last answer token scores higher than MultiHaluDet in all twelve model-setting-label comparisons. Combined with token log-probabilities, it beats MultiHaluDet combined with the same log-probabilities in all eight realistic comparisons, with every confidence interval above zero. MultiHaluDet's training-seed variance (s.d. <=0.005) is far smaller than these gaps. We release our code and evaluation protocol.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-29
DOI
https://doi.org/10.5281/zenodo.23754851
Primary Topic
Adversarial Robustness in Machine Learning
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Answer Length, Not Truth: Re-evaluating White-Box Hallucination Detectors Under Fair Protocols

Mohammed Abdul Jawad Ali Khan
1 links
Zenodo (CERN European Organization for Nuclear Research)
Adversarial Robustness in Machine Learning
preprint

Answer Length, Not Truth: Re-evaluating White-Box Hallucination Detectors Under Fair Protocols

Mohammed Abdul Jawad Ali Khan
preprint en

Abstract

White-box hallucination detectors read a language model's internal states to flag hallucinated answers, and recent work reports AUROC above 98%. We re-evaluate two such detectors, MultiHaluDet and ICR Probe, on Mistral-7B and Llama-3.1-8B under one controlled protocol: shared folds, paired bootstrap confidence intervals, and pass/fail criteria fixed before each experiment. On HaluEval QA, answer length alone reaches 0.97 AUROC; when only answers of equal length are compared, MultiHaluDet falls from 0.985 to 0.805-0.815, and its six-classifier stacking stage adds nothing over averaging its deep models. When the models answer TriviaQA and Natural Questions themselves, MultiHaluDet scores below trained token log-probabilities in seven of eight comparisons and ties in the eighth, and ICR Probe scores below them in all eight. A logistic-regression probe on the full hidden state of the last answer token scores higher than MultiHaluDet in all twelve model-setting-label comparisons. Combined with token log-probabilities, it beats MultiHaluDet combined with the same log-probabilities in all eight realistic comparisons, with every confidence interval above zero. MultiHaluDet's training-seed variance (s.d. <=0.005) is far smaller than these gaps. We release our code and evaluation protocol.

Zenodo (CERN European Organization for Nuclear Research)
Indian Institute of Information Technology Design and Manufacturing, Kurnool (IN)
Adversarial Robustness in Machine Learning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon