Pediatric Nurses’ Perceptions of the Quality and Usefulness of Clinical Summaries: Surgeons Versus Generative AI

Purpose: The purpose of this study was to compare the performance of artificial intelligence (AI)-generated and physician-written handoff summaries in a pediatric surgery population from nurses’ perspective, utilizing a validated quality assessment tool. Methods: Three clinical nurses independently reviewed and compared AI-generated summaries to physician-written summaries for patients admitted to the surgery service line utilizing the validated Physician Documentation Quality Instrument (PDQI-9) tool. The nurse evaluation team met prior to scoring to ensure consistency of interpretation and scoring of PDQI-9 domains. Results: Artificial intelligence-generated and physician written summaries for 117 patients were included, revealing a statistically significant higher ranking for AI-generated summaries (Z = 3.89, p < .001). Artificial intelligence-generated summaries outperformed physician written summaries in seven out of nine PDQI-9 domains (thorough, useful, organized, comprehensible, succinct, synthesized, and internally consistent). Fleiss’ Kappa analysis was conducted to assess the degree of agreement among the raters. The results showed fair agreement for five domains and slight agreement for four domains, highlighting the need and complexity of multirater review of clinical summaries. Practice Implications/Conclusion: This is the first study to examine how nurses evaluate the quality and usefulness of AI-generated summaries in their practice. Artificial intelligence-generated physician summaries improve the quality of handoff for nurses by providing clear and concise information, reducing variability, and providing structure for information sharing. Integration into practice may improve communication, enhance nursing workflow, and improve patient safety. Further research is needed to evaluate AI-generated physician summaries’ long-term effectiveness in patient safety, workload efficiency, and nursing satisfaction.