Answer Length, Not Truth: Re-evaluating White-Box Hallucination Detectors Under Fair Protocols
White-box hallucination detectors read a language model's internal states to flag hallucinated answers, and recent work reports AUROC above 98%. We re-evaluate two such detectors, MultiHaluDet and ICR Probe, on Mistral-7B and Llama-3.1-8B under one controlled protocol: shared folds, paired bootstrap confidence intervals, and pass/fail criteria fixed before each experiment. On HaluEval QA, answer length alone reaches 0.97 AUROC; when only answers of equal length are compared, MultiHaluDet falls from 0.985 to 0.805-0.815, and its six-classifier stacking stage adds nothing over averaging its deep models. When the models answer TriviaQA and Natural Questions themselves, MultiHaluDet scores below trained token log-probabilities in seven of eight comparisons and ties in the eighth, and ICR Probe scores below them in all eight. A logistic-regression probe on the full hidden state of the last answer token scores higher than MultiHaluDet in all twelve model-setting-label comparisons. Combined with token log-probabilities, it beats MultiHaluDet combined with the same log-probabilities in all eight realistic comparisons, with every confidence interval above zero. MultiHaluDet's training-seed variance (s.d. <=0.005) is far smaller than these gaps. We release our code and evaluation protocol.
Authors
- Mohammed Abdul Jawad Ali Khan
Institutions
- Indian Institute of Information Technology Design and Manufacturing, Kurnool (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-29
- DOI
- https://doi.org/10.5281/zenodo.23754851
- Primary Topic
- Adversarial Robustness in Machine Learning
- Type
- preprint