Large Language Models to Summarize Pediatric Admission Notes Into Plain Language

Providing health care information in plain language promotes transparency and improves patient understanding; however, clinical notes are geared toward clinician-to-clinician communication and not in language accessible to patients.1 Large language models (LLMs) can summarize complex medical texts, including radiology reports and discharge summaries, although studies in the pediatric population are limited.2–4The aim of this study was to evaluate Generative Pre-trained Transformer 4 (GPT-4)’s ability to summarize pediatric general care admission notes. We hypothesized that GPT-4 could accurately generate readable and understandable summaries for caregivers, noninferior to those of hospitalists.Sixteen pediatric hospitalists from 2 institutions were recruited to create 20 synthetic admission notes for the 20 most common US pediatric admission diagnoses (Supplemental Table 1).5 They followed a standardized template to create clinically realistic notes while explicitly avoiding patient data and LLM-generated content. Both hospitalists and GPT-4 (OpenAI, March 20–21, 2024) responded to the same optimized prompt to generate summaries, with each hospitalist writing a summary for their synthetic note (Supplemental Tables 2 and 3).2 Summaries were subsequently blinded for evaluation.To assess chatbot accuracy for patient-facing materials, we adapted an evaluation tool rating completeness, error, hallucination, structure, and release readiness on a 5-point Likert scale (Supplemental Table 4).4 Two hospitalists independently scored both hospitalist and GPT-4 summaries, documenting the reasoning behind assigned scores and resolving discrepancies with a third reviewer. Interrater reliability was measured using the intraclass correlation coefficient (ICC).6 Readability was assessed using the Flesch-Kincaid Reading Ease and Grade Level.3 A parent of a child with medical complexity evaluated understandability using the validated Patient Education Materials Assessment Tool (PEMAT).3,7 The institutional review board deemed the study exempt.Analyses were conducted using SAS v9.4 (SAS Institute, Cary, NC). Linear mixed-effects models (LMM) tested differences in readability and understandability. Generalized LMM with a proportional-odds cumulative logit approach tested probability differences in accuracy. Statistical significance was set at P < .05.GPT-4 summaries had better readability scores than hospitalist summaries (Table 1); however, GPT-4 had lower release readiness, a measure of the extent of editing required before providing this summary to a patient (Table 2). Commonly cited reasons include vague details and awkward wording (Supplemental Table 5). There were no differences in completeness, error, hallucination, or structure scores between GPT-4 and hospitalist summaries with good to excellent interrater reliability across all accuracy measures (ICC2,1 = 0.89). Both had similar understandability (least-squares mean [SE]: 57.3 [4.0]% vs 67.5 [4.0]%, P = .08, Table 1).This study compares GPT-4 and hospitalist-generated pediatric admission note summaries while incorporating both a physician-assessed release readiness score and a parent-rated understandability assessment. GPT-4 summaries were easier to read than hospitalist summaries but were less ready for sharing without further editing. These findings highlight the potential of LLMs to draft plain-language summaries of pediatric clinical notes. However, higher readability did not translate to better understandability, because readability formulas do not account for logical flow or coherence. Both GPT-4 and hospitalist summaries scored below the 70% understandability goal, underscoring the challenge of creating summaries that are accessible to patients.3Summarizing pediatric medical text remains complex, requiring communication with both patients and caregivers. This study provides evidence that LLMs produce medically accurate summaries using patient-friendly language. Strengths include the use of a physician-assessed release readiness score and a parent-assessed understandability score.This study has limitations. Synthetic notes were used because GPT-4’s public version does not meet patient privacy standards. Future studies could leverage electronic health record-based LLMs to analyze real clinical notes with larger sample sizes and assess performance differences by age and diagnosis to better interpret CIs. Additionally, no established guidelines define essential content, ideal length, or tone for note summaries.8 Incorporating patient and caregiver perspectives on key summary characteristics can inform LLM training efforts to enhance readability and understandability.9 Although PEMAT is a validated tool, its suitability for patient-facing LLM outputs is unclear. Future studies should include diverse caregivers to improve generalizability, using tools designed to evaluate patient-facing LLM-generated summaries.Given the nuances of clinician-patient communication, clinician review will be required before LLM-generated summaries reach patients or caregivers. With continued refinement and integration into electronic health records, LLMs have the potential to enhance patient understanding. However, careful implementation is necessary to avoid inadvertently increasing clinician workload while meeting the needs of clinicians, patients, and caregivers.