HallucinationNerd: A Framework and Tool to Detect Hallucinations and Unsupported Claims

Retrieval-augmented generation (RAG), agentic systems, and human authors produce text with inline citations, but the cited sources do not always support the claims attributed to them. We present HallucinationNerd, a modular citation-verification engine that decomposes text into atomic claims, retrieves relevant passages from each cited source, and determines whether the source supports the claim. In addition, HallucinationNerd identifies claims that carry no citation and searches user-approved databases to find or refute supporting evidence for them. The system handles five common citation formats, operates on PDFs, web pages, and plain text, and produces per-claim verdicts with evidence pointers. The underlying algorithm mixes parsing (to identify claim–citation pairs), overlapped chunking, stopword elimination, and AI-based semantic comparison. In head-to-head comparison on the same data, HallucinationNerd substantially outperforms (by about 20% in F1 score) state-of-the-art systems on accuracy and precision on a benchmark built from research papers. HallucinationNerd can also be used to verify the output of other AI systems. For example, we audited a set of outputs of Scholar GPT, a publicly available scholarly research agent. In that sample at least, its citations were faithful to the cited sources.