Comparing ICD Codes and Traditional Natural Language Processing to Identify Acute Pediatric Firearm Injury

OBJECTIVES: Firearm injuries are a leading cause of pediatric mortality, but may not be adequately identified by International Classification of Diseases (ICD) codes. Natural language processing (NLP) could improve identification. We derived an NLP model from medical narratives and compared the sensitivity and specificity of ICD codes and the NLP models to identify pediatric acute firearm injury (AFI). METHODS: Retrospective study of ED visits by patients (aged 0 to 17 y) presenting to 7 pediatric emergency departments (ED) with a potential AFI between 2011 and 2019. A medical narrative review was performed to identify AFI narratives. These narratives, plus randomly selected narratives, underwent language feature extraction with logistic regression to identify topics significantly associated with the categories. Two Random Forest models were created: model 1 on all narratives associated with a potential AFI and model 2 using the confirmed AFI narratives. Model 2 was then applied to a new, unreviewed selection of random narratives. A second round of medical narrative review was performed on the random narratives, including those identified by model 2 as AFI. Sensitivity and specificity of ICD and NLP for the identification of AFI were calculated. RESULTS: A total of 2407 ED visits for potential AFI were identified. Of those, 1183 narratives were annotated by narrative review as AFI. In the randomly selected narratives (n=1150), 47 additional AFI were identified, 16 by NLP and 31 by narrative review, for a total of 1230 AFI. ICD codes correctly identified 1171 (95.2%) AFI, with a sensitivity of 95.2% (93.8% to 96.3%) and specificity of 93.5% (92.6% to 94.4%). NLP correctly identified 1041 (84.6%) AFIs in the final data set with a sensitivity of 84.6% (82.5% to 86.6%) and specificity of 94.7% (93.8% to 95.5%). CONCLUSIONS: ICD and NLP both had acceptable sensitivity and specificity for identifying pediatric AFI. Future directions include evaluating newer NLP models to address false positives and the identification of the intent of injury.