S1921 Clear Enough to Drink? A Head-to-Head Comparison of Large Language Model-Generated Colonoscopy Preparation Instructions

Introduction: Patient communication poses a significant challenge in medicine due to literacy gaps and time constraints. Recently, large language models (LLMs) are being used to help with documentation and communication but their medical accuracy and readability vary. High-quality instructions for procedures such as colonoscopy are critical to ensure patient adherence, and safety. This study compared the readability and quality of colonoscopy preparation instructions generated by 5 leading LLMs. Methods: A standardized prompt was used to generate colonoscopy preparation instructions from 5 LLMs: ChatGPT-4, DeepSeek (R1), Gemini 2.5 Flash, Claude Sonnet 4, and Perplexity AI. Outputs were limited to <500 words. Readability was assessed using Flesch Reading Ease Score (FRES), Flesch-Kincaid Grade Level (FKGL), and SMOG Index. Three independent reviewers (RN, EM and JS) evaluated quality using the Patient Education Materials Assessment Tool (PEMAT), the DISCERN score, and a custom Likert-type rubric assessing accuracy, clinical appropriateness, completeness, clarity, harm potential, and bias. Descriptive statistics were used for comparison. Results: All LLMs adhered to the output limit except Gemini (863 words). DeepSeek had the highest FRES (68.2) and lowest FKGL (5), indicating superior readability while Gemini scored the lowest (FRES 59.6, FKGL 7). SMOG scores ranged from 8 (Claude) to 10 (Gemini). All LLMs achieved 100% audience reach and had similar tone profiles. PEMAT scores were consistent across models (actionability 60%, understandability 84.6%). Perplexity scored highest in DISCERN reliability (3.78) due to inclusion of references, while Gemini had the highest quality score (3.22). Overall DISCERN scores ranged 3.17-3.37. ChatGPT had the highest accuracy (4.67), and Gemini provided the most complete content (4.17). Claude was noted for slightly lower harm safety (4) and appropriateness (3.33), while all others scored 4.67. Most models performed well on clarity and neutrality. Conclusion: LLMs can produce colonoscopy preparation instructions with generally acceptable readability and quality. DeepSeek offered the best readability, while Perplexity had better reliability and referencing. ChatGPT generated the most accurate content, and Gemini was the most comprehensive but more complex. Minor safety concerns were noted with Claude. Overall, LLMs are promising tools for patient education but require clinician oversight to ensure safety and suitability for clinical use.