Hybrid retrieval–verification framework for hallucination reduction in medical LLMs

Clinicians use Large Language Models (LLMs) for clinical decision support, medical question answering, and automated document writing. However, these models can hallucinate and pose safety risks. While Retrieval-Augmented Generation (RAG) improves factuality, it does not always produce outputs consistent with supporting evidence. To address these issues, this study reframes NLI-based unsupported-response rate mitigation for generative models as a confidence-aware inference problem and proposes a unified CARE-Med framework for retrieval, generation, and verification. The framework ensures the reliability of its outputs by leveraging dense retrieval, natural language inference verifiers, and a confidence scoring mechanism. The authors validated the anti-NLI-based unsupported-response optimisation method for biomedical large models proposed in this paper against the MIMIC-III, PubMedQA, and BioASQ benchmarks. The method reduces NLI-based unsupported-response rate decreases from 32% for the Baseline LLM to 10% for CARE-Med, corresponding to a 22-percentage-point or 68.8% relative reduction. Compared with conventional RAG, NLI-based unsupported-response rate decreases from 21% to 10%, corresponding to an 11-percentage-point or 52.4% relative reduction. The authors leverage explicit verification and uncertainty reasoning to meet requirements for safe clinical deployment.