SUN-807 Bridging AI and Endocrinology: A Comparative Analysis of Large Language Models in Diabetes Care

Abstract Disclosure: C. Shenoy: None. P. Mehta: None. H. Kreit: None. A. Kumar: None. N. Shah: None. Aims - Artificial intelligence (AI) models like ChatGPT and Google Bard are being more prevalently used in healthcare, however their efficacy compared to physician experts in patient care is still underexplored. This study investigates the performance of large language models (LLMs) in answering diabetes-related questions, compared to an endocrinologist. We evaluate the LLMs’ ability to provide thorough, clear, and simple answers, while exploring physician perspectives on the incorporation of AI in endocrinology. Methods - Based on common patient concerns, we developed 25 standard questions centered on diabetes care. These questions were presented to ChatGPT, Google Bard, and an endocrinology-trained physician, and the generated answers were recorded. Three independent, blinded graders evaluated the responses on a 0-3 scale for thoroughness, clarity, and simplicity. Intra- and inter-grader variability was assessed using Wilcoxon Rank Sum tests. Written physician feedback on the role of AI in endocrinology was analyzed to identify key insights on its applications and limitations. Results - Across all models, ChatGPT achieved the highest clarity score (mean=2.68, SD=0.55), excelling in patient education and prevention-related topics. However, its thoroughness was significantly variable between graders (P<0.05), limiting its reliability for complex clinical scenarios. Google Bard scored the highest in simplicity (mean=2.84, SD=0.62), illustrating its effectiveness for concise, patient-friendly tools like glucose logs and meal plans. The endocrinologist provided the most thorough responses (mean=2.96, SD=0.69, P<0.0001 vs. AI models) but reported the lowest simplicity score (mean=1.92, SD=0.99), reflecting a focus on clinical depth over accessibility. Physician feedback highlighted AI’s potential to simplify complex concepts and offer practice guidance, but also raised concerns about accuracy, overreliance, and data privacy. While AI models were seen as valuable supplements, they are not replacements for physician expertise. Conclusions - LLMs show early promise in advancing patient education, especially with strategies for disease prevention. While ChatGPT excels in clarity and Bard in simplicity, their limited consistency and depth underscore the need for cautious integration alongside physician expertise. Future efforts should focus on developing hybrid AI models that combine the strengths of LLMs, with the nuanced decision-making of healthcare providers. To increase equitable and safe LLM integration, future clinical research needs to establish robust ethical AI guidelines, refine AI training to reduce response variability, incorporate patient perspectives, and conduct studies across diverse settings with larger cohorts. Presentation: Sunday, July 13, 2025