Conformal selective answering with hidden-state probes controls FDR on HaluEval but fails under domain shift in small language models

Whether the hidden state probe can provide a false discovery rate (FDR) guarantee for the answer release of the small language model still lacks a cross-domain shift test. This article uses a multi-layer probe to identify the hallucinated answer, decides to release it with conformal selection, and tests it on HaluEval and TruthfulQA. The probe AUROC (area under the ROC curve) on HaluEval is 0.991, which is higher than 0.974 of the answer-length baseline; the FDR at the target level of 0.10 is 0.050, and the power is 0.992, which is higher than the baseline of 0.883. The exploratory AUROC with a single feature of punctuation at the end of the sentence reached 0.969, and the difference in the length and sentence style of HaluEval's answer contributed most of the AUROC. When cross-domain, AUROC dropped to 0.491, and FDR rose to 0.409, exceeding the target level; after recalibration, 0 answers were released. Exploratory in-domain retraining brings AUROC back to 0.904, FDR is 0.045, and power is 0.502. High AUROC cannot be directly interpreted as the model grasping the true and false information, and the probe must be retrained with the data of the target domain.

Authors

Institutions

Publication Details

Journal
Advances in Engineering Innovation
Published
2026-11-16
DOI
https://doi.org/10.54254/2977-3903/2026.39108
Primary Topic
Topic Modeling
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Conformal selective answering with hidden-state probes controls FDR on HaluEval but fails under domain shift in small language models

Tingzhang Yang, Yulin Tu, Fan Si, Xing Li
Advances in Engineering Innovation
Topic Modeling
article

Conformal selective answering with hidden-state probes controls FDR on HaluEval but fails under domain shift in small language models

Tingzhang Yang, Yulin Tu, Fan Si, Xing Li
article en

Abstract

Whether the hidden state probe can provide a false discovery rate (FDR) guarantee for the answer release of the small language model still lacks a cross-domain shift test. This article uses a multi-layer probe to identify the hallucinated answer, decides to release it with conformal selection, and tests it on HaluEval and TruthfulQA. The probe AUROC (area under the ROC curve) on HaluEval is 0.991, which is higher than 0.974 of the answer-length baseline; the FDR at the target level of 0.10 is 0.050, and the power is 0.992, which is higher than the baseline of 0.883. The exploratory AUROC with a single feature of punctuation at the end of the sentence reached 0.969, and the difference in the length and sentence style of HaluEval's answer contributed most of the AUROC. When cross-domain, AUROC dropped to 0.491, and FDR rose to 0.409, exceeding the target level; after recalibration, 0 answers were released. Exploratory in-domain retraining brings AUROC back to 0.904, FDR is 0.045, and power is 0.502. High AUROC cannot be directly interpreted as the model grasping the true and false information, and the probe must be retrained with the data of the target domain.

Advances in Engineering InnovationVol. 17(12)
Chongqing Technology and Business University (CN), Yonsei University (KR), Peking University (CN)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon