S2961 Assessing the Quality and Readability of AI Chatbot-Generated Patient Education Materials for Endoscopic Retrograde Cholangiopancreatography

Introduction: Artificial intelligence (AI), particularly large language model chatbots, is increasingly used to generate patient education materials (PEM). While AI-generated PEM is generally high quality, its readability is often poor. The utility of AI-generated PEM for Endoscopic Retrograde Cholangiopancreatography (ERCP) has not been thoroughly investigated. This study aims to evaluate the quality and readability of AI chatbot-generated PEM for ERCP. Methods: Ten questions related to ERCP were formulated based on existing website PEM. These questions were submitted sequentially to 4 widely used chatbots: ChatGPT-4o, Microsoft Copilot, Google Gemini, and Meta AI. Readability was assessed using the Flesch Reading Ease Score (FRES) and 6 grade-level metrics: Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, Simple Measure of Gobbledygook, Automated Readability Index, and FORCAST. Quality was evaluated using 2 validated PEM tools: a modified DISCERN questionnaire and a modified Patient Education Materials Assessment Tool (PEMAT). Results: The mean FRES across all chatbots was 32.16. Individual FRES scores were 39.45 for ChatGPT, 40.3 for Microsoft Copilot, 38.08 for Google Gemini, and 10.81 for Meta AI. The overall mean grade levels across the 6 readability tools were 13.37 (Flesch-Kincaid Grade Level), 15.06 (Gunning Fog Index), 15.52 (Coleman-Liau Index), 15.31 (Simple Measure of Gobbledygook), 14.24 (Automated Readability Index), and 11.99 (FORCAST). Mean reading grade levels for each chatbot across the 6 tools were 13.69 for ChatGPT, 13.37 for Microsoft Copilot, 12.94 for Google Gemini, and 16.98 for Meta AI. In the reliability section of the DISCERN questionnaire, all chatbots scored 4.6, except for Google Gemini, which scored 4.4. For the quality section of the DISCERN questions, ChatGPT scored 4.29, Meta AI 4.14, Microsoft Copilot 4.0, and Google Gemini 3.57. Regarding PEMAT scores, ChatGPT and Google Gemini achieved 100% for understandability, while Microsoft Copilot and Meta AI scored 91.6%. All chatbots scored 75% for the actionability portion of the PEMAT questionnaire. Conclusion: This study shows AI chatbots can generate high-quality PEM for ERCP, but readability is poor. For better patient understanding and adherence, chatbot-generated PEM should aim for a sixth-grade reading level. However, responses averaged about twice the target level. While chatbot use is not discouraged, providers should inform patients of limits in readability. Future research should develop and test AI tools engineered for patient-centric language.