Hallucination Detection Using Evidence Retrieval and Natural Language Inference

Abstract Large language models (LLMs) can generate fluent responses containing unsupported, incorrect, or unverifiable factual statements. TruthLens is an evidence-based claim-verification pipeline that examines factual statements individually rather than assigning one label to an entire answer. It extracts sentence-level claims, retrieves candidate evidence using BM25 lexical retrieval and MiniLM semantic retrieval, combines rankings with Reciprocal Rank Fusion (RRF), and applies a pretrained DeBERTa-v3 Natural Language Inference (NLI) model to the highest-ranked evidence sentence. Entailment, contradiction, and neutral predictions are mapped to SUPPORTED, CONTRADICTED, and UNKNOWN. On 19,998 FEVER development examples and a restricted 27,055-sentence evidence corpus, hybrid retrieval achieved Recall@1/5/10 of 56.51%/65.02%/65.67%, outperforming the individual retrieval methods in this experiment. A frozen hybrid top-1 NLI configuration achieved 62.25% accuracy and 0.6231 macro-F1 on a separate 2,000-example held-out subset. An exploratory HaluEval experiment showed poor transfer under the current question-answering evaluation formulation. The results demonstrate a working retrieval-and-verification prototype while exposing limitations in evidence coverage, claim extraction, coreference, uncertainty handling, and cross-dataset generalization.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11701365/v1
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Hallucination Detection Using Evidence Retrieval and Natural Language Inference

Mayank Goswami, Krish
Research Square
Topic Modeling
preprint

Hallucination Detection Using Evidence Retrieval and Natural Language Inference

Mayank Goswami, Krish
preprint en

Abstract

Abstract Large language models (LLMs) can generate fluent responses containing unsupported, incorrect, or unverifiable factual statements. TruthLens is an evidence-based claim-verification pipeline that examines factual statements individually rather than assigning one label to an entire answer. It extracts sentence-level claims, retrieves candidate evidence using BM25 lexical retrieval and MiniLM semantic retrieval, combines rankings with Reciprocal Rank Fusion (RRF), and applies a pretrained DeBERTa-v3 Natural Language Inference (NLI) model to the highest-ranked evidence sentence. Entailment, contradiction, and neutral predictions are mapped to SUPPORTED, CONTRADICTED, and UNKNOWN. On 19,998 FEVER development examples and a restricted 27,055-sentence evidence corpus, hybrid retrieval achieved Recall@1/5/10 of 56.51%/65.02%/65.67%, outperforming the individual retrieval methods in this experiment. A frozen hybrid top-1 NLI configuration achieved 62.25% accuracy and 0.6231 macro-F1 on a separate 2,000-example held-out subset. An exploratory HaluEval experiment showed poor transfer under the current question-answering evaluation formulation. The results demonstrate a working retrieval-and-verification prototype while exposing limitations in evidence coverage, claim extraction, coreference, uncertainty handling, and cross-dataset generalization.

Research Square
Indian Institute of Technology Roorkee (IN)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon