Artificial intelligence vs. physicians: Quality of oncology patient education materials.

408 Background: ChatGPT, a large language model (LLM) chatbot, emerged as a significant source of information for patients and providers. As artificial intelligence (AI) continues to evolve, its integration into educational content creation offers promising enhancements in the delivery of medical information. This study compares the effectiveness and readability of AI-generated and physician-generated educational materials in oncology, focusing on various symptoms and side effects of treatment. Methods: Using a standardized prompt, educational materials on six side effects of treatment were generated by ChatGPT (version GPT-4o) and separately by a group of three physicians. These topics included cytopenia, diarrhea/constipation, fatigue, nausea, neuropathy, and loss of appetite. Eight hematology-oncology providers, blinded to the source of the materials, independently scored them using a 5-point Likert scale across six categories adapted from the Patient Education Material Assessments Tool (PEMAT) for Printable Materials: content, organization, accuracy, clarity, engagement, and actionability. Readability was assessed using the Flesch-Kincaid tests. Preferences for the most effective educational material and perceptions of its source were also recorded. The Mann-Whitney U test was applied to assess statistical differences between the two groups. Results: Both AI- and physician-generated materials received consistent median scores of 4.0 across all categories, indicating a generally high level of perceived quality. Mann-Whitney U tests showed no significant differences between the two groups. The Flesch-Kincaid grade levels were 11.90 for AI and 11.52 for the physician group, and the Flesch Reading Ease scores were 34.28 for AI and 38.17 for the physician group, showing a non-significant trend towards easier readability in the physician group. Preferences slightly favored physician materials (54.17% vs. 45.83% for AI), and AI content was correctly identified 35.42% of the time. Comments from evaluators indicated that AI-generated materials were more concise and organized but had a rare inaccuracy in medication frequency instructions. Conclusions: This study highlights the efficacy of AI, specifically ChatGPT-4, in creating oncology patient education materials comparable to those by physician experts. Evaluator preferences slightly favored physician materials, but AI content was recognized as concise and organized. These findings suggest that tools like ChatGPT, when augmented by physician review, can serve as valuable resources for creating accurate and relevant educational content in oncology.