A Systematic Comparative Analysis of RAGAS versus LLM-as-Judge for Generative AI Factuality Assessment

As large language models (LLMs) are increasingly considered for tasks that involve communicating health and regulatory information to the public, the risk that these models fabricate, conflate, or misrepresent critical information poses a direct threat to safe and reliable deployment. Semi-automated methods for verifying the factuality of generative AI content are therefore essential before such tools can support operational activities at agencies like the U.S. Food and Drug Administration (FDA). To evaluate whether factuality can be assessed at scale, we conducted a comparative analysis of two LLM-based evaluation methods, RAGAS (Retrieval-Augmented Generation Assessment Suite) and LLM-as-Judge, to assess their ability to detect errors in retrieval-augmented generation (RAG) systems. Using expert-written responses to general public inquiries drafted by FDA subject matter experts, we simulated AI hallucinations by manually editing these ground-truth responses under a controlled design that varied both the type and severity of factual error. Corruptions applied to official responses were categorized as non-substantive (no impact on meaning) or substantive (impact on meaning) and stratified by increasing levels of lexical modification (minor, moderate, major). Both evaluation methods demonstrated an ability to differentiate between substantive and non-substantive changes; however, we found substantial overlap in scores, preventing the use of a simple cutoff for labeling substantive changes. A logistic regression model using factuality scores from both methods as features achieved better performance than classifying based on one set of scores alone (79% accuracy, 96% recall, 82% F1-Score), missing only 4% of substantive cases. The combined approach leveraging RAGAS and LLM-as-Judge methods proved most effective for hallucination detection, demonstrating that ensemble evaluation methods can improve identification of substantive content modifications in LLM responses. These findings indicate that RAGAS and LLM-as-Judge, particularly in combination, can detect hallucinations in complex, domain-specific RAG applications, while the substantial overlap in individual scores highlights the limits of using either method's scores as standalone quality measures. Ensemble factuality evaluation could serve as an automated safety check within deployed health-information RAG systems, reducing reliance on manual expert review.