HalluSense: A Span-Level Hallucination Detection
Many question-answering systems first look up passages in a document collection and then write their reply fromwhat they found. This design is known as Retrieval-AugmentedGeneration (RAG), and it is meant to keep answers tied to realsources. Even so, such systems still produce smooth sentencesthat the retrieved text does not support. Existing detectorsusually return one verdict for the entire answer, and they saynothing about the reason for the failure, which may lie in thesearch step, in the language model, or in both. We presentHalluSense, an auditing layer placed on top of a RAG pipeline.It cuts the answer into atomic claims, checks every claim againstthe retrieved text with an NLI cross-encoder, calibrates thescores with isotonic regression, and labels each flagged claimas retrieval-caused, generation-misstatement or ungrounded. AStreamlit dashboard shows the evidence, a retrieval-rank trace,a fix hint and a trust score, while a FastAPI service keeps everyaudit in SQLite. On a held-out RAGTruth sample the calibrateddetector reaches a sentence-level AUROC of 0.816 and lowers theexpected calibration error from 0.954 to 0.045. On a 150-caseattribution benchmark it reaches 83.3% three-class and 84.0%two-class accuracy. A full audit takes about 16 seconds on alaptop, and nearly all of that time goes to the local languagemodel.
Authors
- Yash Yash Sureja
- Manthan Manthan Sasane
- Shravan Shravan Mehta
- Rohan Rohan Divekar
- Rupali Bhatkhande, Rupali Ma'am
Institutions
- MIT Art, Design and Technology University (IN)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24349104
- Primary Topic
- Topic Modeling
- Type
- preprint