S500 AI-Care: Comparative Evaluation of ChatGPT, Copilot, and Gemini for Post-Colonoscopy Patient Education

Introduction: Artificial intelligence (AI) chatbots are increasingly used by patients seeking medical information, particularly after procedures like colonoscopy. ChatGPT has shown the ability to generate accurate, accessible responses to gastrointestinal questions, sometimes outperforming traditional health websites. However, comparative data on leading platforms: ChatGPT (OpenAI), Copilot (Microsoft), and Gemini (Google) remain limited. We conducted a structured evaluation of these chatbots’ responses to 8 common post-colonoscopy questions, rated by GI fellows and attendings using a 5-point Likert scale. Methods: Eight common post-colonoscopy questions were sourced from FAQs of the top 50 U.S. hospital websites and submitted verbatim to ChatGPT, Copilot, and Gemini. Responses were anonymized and independently rated by 6 GI fellows and 3 attending physicians, blinded to chatbot source (single-blind). Reviewers scored each response across 4 domains: accuracy, clarity, empathy, and usefulness. Mean scores were calculated by domain and tool; analysis of variance (ANOVA) and pairwise t-tests were used for comparison. Results: Nine evaluators (3 attendings, 6 fellows) rated chatbot responses to 8 post-colonoscopy questions across 4 domains. ChatGPT achieved the highest mean score for usefulness (3.71), Copilot for clarity (3.69), and Gemini for accuracy (3.61); empathy consistently rated lowest across platforms, with Gemini receiving the lowest score (3.26). Despite domain-specific variation, no statistically significant differences were found in overall performance (ANOVA P = 0.677; all pairwise P > 0.4). Attendings and fellows showed strong inter-rater agreement and consistent ranking patterns. These findings suggest comparable overall quality among the chatbots, with minor variation in communication attributes relevant to patient care. Conclusion: This study demonstrates that leading AI chatbots can generate clinically appropriate, patient-friendly responses to routine post-colonoscopy questions. ChatGPT and Copilot, both GPT-4–based, performed slightly better overall, especially in clarity and usefulness. However, limitations in empathy suggest that while chatbots may assist in patient education, they cannot yet replicate the nuance of human interaction. Consistent evaluator scores across training levels support reproducibility and generalizability. These findings support AI chatbots as adjuncts to post-procedure care, with appropriate oversight to ensure safe and effective communication.