Evaluating the Quality and Readability of AI-Generated Patient Education Materials for Hysterectomy [ID 1378]

INTRODUCTION: Surgical counseling for hysterectomy can overwhelm patients, making supplemental materials valuable. We evaluated the quality and readability of hysterectomy educational materials generated by three AI chatbots. METHODS: We prompted three AI chatbots—ChatGPT, Perplexity, and Microsoft Bing AI—to answer questions from the UpToDate patient handout “Hysterectomy: (The Basics)” for comparison. Three independent scorers evaluated quality using the DISCERN instrument. Readability was assessed using the Flesch–Kincaid Grade Level (FKGL), Flesch–Kincaid Reading Ease (FKRE), and Gunning Fog Index (GFI). RESULTS: The mean UpToDate DISCERN score was 3.7 (SD 0.58) compared to the mean DISCERN score for the chatbots of 3.25 (SD 0.71). The mean chatbot FKRE scores were 29.18 (SD 4) (college graduate-level). UpToDate’s mean FKGL was 6.9 (SD 0.06), indicating a middle school reading level, whereas the chatbots scored 13.4 (ChatGPT), 11.9 (Perplexity), and 12.6 (Microsoft), corresponding to college level. The mean GFI was 9.1 (SD 1.0) for UpToDate (grade 9), and the mean was 20.2 (SD 8.3) (university level) for the chatbots. CONCLUSIONS/IMPLICATIONS: According to the DISCERN tool, the quality of AI-assisted and UpToDate patient information on hysterectomy is of medium quality. Queries generated by chatbots demonstrate higher reading levels and exhibit lower reliability and accuracy than corresponding information from UpToDate. Due to the nascent nature of this technology, AI-generated content on hysterectomy should not be relied upon as a direct resource for patient care.