Validation of a Natural Language Processing-Assisted Chart Review Method for Identifying Advance Care Planning Documentation in Pediatric Critical Illness

OBJECTIVES: Reliable measurement of advance care planning (ACP) is essential for evaluating both communication practices and quality of care in pediatric critical illness. However, most ACP documentation is embedded in free-text clinical notes and cannot be captured using administrative data, while manual chart review is time-intensive and difficult to scale. We sought to validate a rule-based natural language processing (NLP) approach to identify ACP documentation in the electronic health record (EHR) compared with manual chart review. DESIGN: Retrospective cohort study. SETTING: Single-center, quaternary PICU. PATIENTS: Children younger than 21 years admitted to the PICU for greater than 24 hours following out-of-hospital cardiac arrest (OHCA) from 2012 to 2024. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: A rule-based NLP approach incorporating semi-automated chart review was developed through iterative refinement of previously validated keyword libraries to identify ACP documentation across prespecified domains in the EHR. In the validation cohort ( n = 95), NLP-assisted chart review showed complete agreement with manual chart review (F1 = 1.0) while substantially reducing abstraction time, enabling analysis of over 31,479 clinical notes in 3 days compared with 6 months for manual review of 43,179 notes. Among 125 children with OHCA, 65% had documented ACP discussions ( n = 81). Goals-of-care conversations were most common (99%), followed by limitations of life-sustaining treatment (78%), subspecialty palliative care involvement (28%), preferred location of death (9%), and hospice discussions (7%). CONCLUSIONS: NLP-assisted chart review can efficiently and accurately identify ACP documentation in pediatric critical illness for retrospective research and quality improvement efforts. This semi-automated workflow substantially reduced manual abstraction burden while maintaining excellent agreement with manual chart review, enabling effective abstraction of communication processes not captured in structured data.