Comparing the Readability and Content Quality of Online Patient Education Materials and ChatGPT-Generated Patient Education Materials for Breast Cancer Surgery and Reconstruction

Abstract Patients undergoing breast cancer surgery and reconstruction seek information using online patient education materials (OPEMs). The National Institutes of Health (NIH) and American Medical Association (AMA) recommend a sixth-grade reading level for OPEMs. In recent years, Chat Generative Pre-Trained Transformer (ChatGPT), a large language model (LLM), has shown potential utility in patient education. This study compares the readability and content quality of OPEMs on breast cancer surgery and reconstruction with ChatGPT-generated materials. Google searches were conducted in January 2025 to identify relevant OPEMs for breast cancer surgery and reconstruction. For each search term, ChatGPT 4.0 was prompted to generate patient education guides using two approaches: (1) Standard prompting and (2) simplified prompting to align with NIH/AHA recommendations (“write the guide like I am in sixth grade”). Readability and content quality metrics were assessed. Ninety-nine OPEMs and 60 ChatGPT responses (30 standard, 30 simplified) were analyzed. Median Flesch–Kincaid Grade Level (FKGL) was 10.8 for OPEMs, 10.0 for standard ChatGPT responses, and 5.8 for simplified ChatGPT responses. OPEMs and standard ChatGPT responses significantly exceeded NIH/AMA recommendations (p < 0.001). Simplified ChatGPT responses aligned with the sixth-grade level and were significantly easier to read than OPEMs and standard ChatGPT responses (p < 0.001). DISCERN scores did not significantly differ between OPEMs and standard/simplified ChatGPT responses. OPEMs on breast cancer surgery and reconstruction exceed recommended readability levels. ChatGPT, when prompted to simplify, produced materials consistent with NIH/AMA guidelines while maintaining content quality. Using ChatGPT for patient education may enhance accessibility and patient comprehension of health information.