2060-LB: Benchmarking AI in Type 1 Diabetes Management—How Well Do Generative AI Platforms Follow ADA SOC?

Introduction and Objective: With the rise of generative AI in healthcare, this study evaluates the internal consistency (within platforms) and external consensus (among the platforms) of diabetes care following the ADA SOC generated by AI platforms (OpenAI ChatGPT-4, Google Gemini, Copilot, Perplexity) and compare their responses with clinicians’ notes. Methods: Eight de-identified clinical cases with type 1 diabetes were extracted from a database and edited for clarity. Cases were input into the platforms, generating seven rounds of responses per case per platform. Outputs were analyzed across seven themes: Glycemic management; Lifestyle recommendations; Patient education; Psychosocial considerations; Preventative screenings and immunizations; Patient-specific considerations; and Social determinants of health. Results: Conclusion: The AI platforms showed various levels of internal consistency in their responses, with the highest consistency in CGM patient education. For external consensus among all platforms, the theme with the closest scores was CGM patient education; however, the theme with the highest Jaccard Similarity Score was Discussing Insulin Treatments. AI-generated guidance diverged markedly from clinicians’ recommendations. While more research is needed, these findings emphasize the potential of AI in supplementing diabetes care and highlight the need for human oversight to ensure comprehensive, patient-centered management. Disclosure G. Miranda: None. S.M. Wong: None. C.F. Young: Consultant; Sanofi.