Towards automatic analysis of clinical notes for injuries of bicycles and other micromobility devices using natural language processing

Background Micromobility-related injuries are increasing, yet injury surveillance systems lack comprehensive data on risk factors like helmet use. Unstructured clinical notes in electronic health records contain this valuable information, which is crucial for injury prevention efforts. Methods We conducted a retrospective method comparison study comparing rule-based systems and machine learning models (Term Frequency-Inverse Document Frequency+eXtreme Gradient Boosting) for extracting helmet use information from emergency department triage notes. We analysed 1206 notes from micromobility-related injuries collected between 2018 and 2023 from three institutions: University of Washington Medicine facilities, Washington State Department of Health’s Rapid Health Information NetwOrk system and the US Consumer Product Safety Commission’s National Electronic Injury Surveillance System. Performance was evaluated using F1 score, precision and recall, with both internal and external validation (across three institutions) to assess generalisability. Results The rule-based system outperformed machine learning approaches, achieving F1 scores of 0.96 and 0.95 in two datasets and 0.86 in a third dataset, compared with machine learning models’ F1 scores of approximately 0.80. The rule-based approach demonstrated higher precision and recall while requiring fewer computational resources. External validation confirmed the robustness of the rule-based system with minimal performance degradation across all three institutions. Discussion Despite the current emphasis on machine learning approaches for clinical text analysis, our findings indicate that rule-based systems remain effective for helmet use identification. Conclusion For extraction tasks such as helmet use identification from triage notes, rule-based systems offer superior performance, reliability and efficiency compared with more complex machine learning models. These findings have implications for injury surveillance and prevention programme development.