Agentic AI-Generated vs. Clinician-Authored Hospital Course Narratives for Cardiology Hospitalizations: A Blinded Physician Evaluation

Background: Discharge summaries are essential for safe transitions of care, but writing the hospital course narrative is time intensive. Large language models can generate plausible clinical narratives, yet prior approaches often rely on static inputs and remain vulnerable to omissions and factual inconsistencies. Methods: We developed an electronic health record (EHR)-integrated agentic artificial intelligence (AI) system that uses Gemini 2.5 Pro and Model Context Protocol tools to retrieve clinical notes, diagnostic reports, diagnoses, medications, and appointments on demand. In this retrospective, blinded, paired study, 50 inpatient cardiology encounters were randomly selected from admissions July 13 to 27, 2024, at Mayo Clinic, Rochester, Minnesota. Five physicians rated AI-generated and clinician-authored hospital course narratives for each encounter (250 ratings) on 0-to-10 scales for overall quality, clarity, completeness, accuracy, and readability and indicated a preference. The primary outcome was the paired difference in overall quality (linear mixed-effects model with crossed random intercepts for encounter and reviewer). Secondary domains were analyzed similarly with Bonferroni adjustment, and preference with a crossed random-effects logistic model. Results: Clinician-authored narratives were preferred in 132 ratings (52.8%) and AI-generated narratives in 118 (47.2%) (odds ratio for clinician-authored preference, 1.12; 95% CI, 0.71 to 1.78). Overall quality did not differ significantly (mean, 7.27 AI-generated versus 7.42 clinician-authored; difference, -0.15; 95% CI, -0.86 to 0.56; p=0.627). AI-generated narratives were rated higher for clarity (0.36; 95% CI, 0.07 to 0.65) and readability (0.72; 95% CI, 0.20 to 1.24) but lower for completeness (-0.84; 95% CI, -1.36 to -0.32) and accuracy (-0.35; 95% CI, -0.66 to -0.05). After Bonferroni correction, only lower completeness remained significant (adjusted p=0.012). Conclusions: Hospital course narratives generated by the EHR-integrated agentic AI system did not differ significantly from clinician-authored narratives in physician-rated overall quality or preference. Lower completeness supports continued clinician review before use in the medical record.