Large Language Model–Assisted Personalized Patient Education for Chronic Atrophic Gastritis: A Blinded Paired Clinical Informatics Evaluation

Abstract Background Chronic atrophic gastritis (CAG) is a gastric precancerous condition requiring individualized communication regarding endoscopic and histopathological findings, Helicobacter pylori status, cancer risk, treatment, and surveillance. Large language models (LLMs) may facilitate individualized patient education, but evidence evaluating their performance within clinically matched and blinded assessment frameworks remains limited. Objective To evaluate the quality of large language model–assisted personalized patient education materials for chronic atrophic gastritis using a blinded clinical informatics framework and to compare their performance with clinician-generated materials. Methods In this single-center, blinded, paired comparative study, 50 patients with CAG were used to generate 200 patient education materials: one each from GPT-6.1 Sol, Qwen3.8 Flash, and two gastroenterologists. All sources received identical deidentified clinical information and standardized generation constraints. Three independent gastroenterology experts evaluated all materials using the 19-item Chronic Atrophic Gastritis Patient Education Quality Scoring System (CAG-PEQS), which assessed medical accuracy, personalization, comprehensiveness, and safety (overall score range, 0–95). Objective linguistic and structural characteristics were analyzed separately. Between-source comparisons were performed using Friedman tests followed by Holm-adjusted Wilcoxon signed-rank tests, and inter-rater reliability was assessed using intraclass correlation coefficients. Results Overall CAG-PEQS scores differed significantly among Qwen3.8 Flash, GPT-6.1 Sol, and the clinician reference (Friedman test, P < 0.001). Qwen3.8 Flash and GPT-6.1 Sol achieved high and comparable overall scores, with median scores of 82.67 (IQR 81.17–83.67) and 82.50 (IQR 81.42–83.58), respectively, whereas the clinician reference showed a lower median score of 52.58 (IQR 49.04–55.04). Both LLM-generated materials demonstrated higher scores than the clinician reference across all four CAG-PEQS domains, including medical accuracy, personalization, comprehensiveness, and safety. Inter-rater reliability for the overall CAG-PEQS score was high (ICC = 0.917, 95% CI 0.902–0.929). Objective linguistic analyses showed that both LLMs generated longer materials than the clinician reference, with median Chinese character counts of 405.5 for Qwen3.8 Flash, 395.0 for GPT-6.1 Sol, and 293.0 for the clinician reference. Conclusions Under standardized case-based conditions, LLM-generated CAG patient education materials achieved higher expert-rated quality scores than clinician-generated reference materials. These findings suggest that LLMs may have potential as clinician-supervised tools for drafting structured patient education materials. However, the observed differences in expert ratings do not establish superiority in real-world physician–patient communication or improvements in patient comprehension and clinical outcomes. Further studies incorporating patient-centered assessments, validated readability measures, and real-world clinical evaluation are needed to establish their clinical utility.