SUN-806 Do Customized Chatbots Help with Diabetes Management?

Abstract Disclosure: A. Shams: None. G. Wu: None. E. Pan: None. P. Badala: None. I. Chim: None. R. Toram: None. V.V. Toram: None. B. Hoang: None. Background: As of 2020, around 34 million US adults had diabetes, and 88 million had pre-diabetes. 35% are compliant with diabetes medications or diet. There are 123.5 million active users of ChatGPT in 2024. 8.5 billion searches on Gemini per day. 7% of searches on Google are for health-related questions, or 70,000 per minute.The LLMs such as ChatGPT and Gemini have allowed free or low-cost application programming interfaces (API) for all students, programmers, and scientists to access and modify user prompts to further the adoption of LLMs. Purpose: Are Customized LLMs helpful to our Diabetic patients? Methods: LLMs with preprogrammed APIs were asked the question, “What are the risks of unmanaged diabetes?” The output of each LLM was copied into Gemini 1.5 Pro, and the model was asked: “Evaluate and summarize the differences between each agent and the quality of the responses.” Gemini’s opinions on the performance of each chatbot were analyzed, and discrepancies and findings were noted. Results: Between the 9 models (GPT-4o, GPT-4, GPT-4o-MINI, GPT-4 Turbo, GPT-3.5 Turbo, Mistral 7b, Gemini 2.0 Flash, Gemini 1.5 Flash, and Gemini 1.5 Pro) the most cost-efficient models were GPT-4o-MINI and GPT-3.5 Turbo. These LLMs offer a great cost-to-value ratio for concise but accurate outputs. The best in depth were Gemini 2.0 Flash and GPT-4 Turbo which stood out for detailed insights but with different costs. The free Gemini models provide comprehensive answers at no cost but may use higher tokens ( 1 token = 1 word).Using the Endocrine Society as a “Foundation Model” or “standard, Retrieval-augmented generation (RAG) based responses emphasized key complications like blindness and kidney failure. Direct references to factors such as dyslipidemia and high blood pressure further strengthened the responses. LLMs, particularly GPT-4o, Gemini 2.0 Flash, and GPT-4 Turbo, offered detailed breakdowns of complications, such as mental health impacts, infections, cognitive decline, and acute conditions (DKA), but fell short in emphasizing the prevalence of diabetes as a leading cause of severe outcomes. Overall, RAG responses excel in grounded authority and LLMs add comprehensive detail. Conclusion: All models provided accurate information on the major complications that could arise from unmanaged diabetes, and the models also all provided information about the importance of managing diabetes to reduce these risks. The prioritization of complications is not as clear as a human interaction with a physician, but with APIs, the LLM responses are information. Presentation: Sunday, July 13, 2025