Abstract 4147439: Accuracy of Rule-based Natural Language Processing Models for Identification of Pulmonary Embolism

Background: Some rule-based natural language processing (NLP) tools can identify pulmonary embolism (PE) from imaging reports. The external validity of these tools remains to be determined. Methods: We evaluated data for hospitalized patients from the Mass General Brigham (MGB) Health System (2016-2021). The NLP algorithms developed by Verma et al. and Johnson et al. were implemented on radiology reports to identify PE among 1,712 patients (mean age 60.6 years, 52.3% female). Previously, these two models had demonstrated high diagnostic accuracy for PE internally within the derivation cohorts (Figure). Manual chart review by two independent physicians using pre-specified criteria was set as the reference standard. Weighted estimates were determined based on the total number of hospitalizations at MGB (n=381,642). Results: The prevalence of PE in the weighted sample was 2.0% (7,708/ 381,642). The weighted estimates indicated high sensitivity (77.7% and 88.1%) and specificity (98.0% and 97.2%) but low positive predictive values (44.3% and 39.7%) for the models by Verma et al. and Johnson et al. (Figure). The most frequent reason for false positive findings was the inability of the tools to properly detect negation (such as “no PE”) from the radiology reports. Most false negative findings were related to the patients diagnosed with PE at another hospital and transferred to MGB; subsequently, they did not have an available imaging report from the index hospitalization. Therefore, the NLP algorithms led to false negative results (Figure). Conclusions: The rule-based NLP models showed high sensitivity and specificity for PE detection. However, they resulted in low positive predictive value in external validation, contrary to the derivation cohorts, limiting their utilization. Modifying the existing NLP tools and using large language models should be considered in upcoming studies to improve the positive predictive value.