ID: 4349993 QUALITATIVE EVALUATION FRAMEWORK FOR COMPARING THE EFFECTIVENESS OF LARGE LANGUAGE MODELS THAT POWER HEALTH CARE CONVERSATIONS USING GENERATIVE ARTIFICIAL INTELLIGENCE IN ATRIAL FIBRILLATION

Generative AI (GenAI) is employed across industries, including healthcare, where hallucinations (fabricated or incorrect information) can have dangerous consequences. Existing evaluation metrics for large language models (LLMs) are primarily generic, emphasizing correctness without addressing the knowledge and conceptual nuances necessary in healthcare. This highlights the need for a tailored evaluation benchmark to ensure outputs are Accurate, Relevant, Trustworthy: Fair, Robust, Explainable, Equitable—F.R.E.E, Empathy, Safe —A.R.T.E.S, and efficient for clinical use.