A comparative performance analysis of regular expressions and a large language model-based approach to extract the BI-RADS score from radiological reports.

Background Different natural language processing (NLP) techniques have demonstrated promising results for data extraction from radiological reports. Both traditional rule-based methods like regular expressions (Regex) and modern large language models (LLMs) can extract structured information. However, comparison between these approaches for extraction of specific radiological data elements has not been widely conducted. Methods We compared accuracy and processing time between Regex and LLM-based approaches for extracting Breast Imaging-Reporting and Data System (BI-RADS) scores from 7764 radiology reports (mammography, ultrasound, MRI [magnetic resonance imaging], and biopsy). We developed a rule-based algorithm using Regex patterns and implemented an LLM-based extraction using the Rombos-LLM-V2.6-Qwen-14b model. A ground truth dataset of 199 manually classified reports was used for evaluation. Results We did not detect a statistically significant difference in the accuracy in extracting BI-RADS scores between Regex and an LLM-based method (accuracy of 89.20% for Regex vs 87.69% for the LLM-based method; P = .56, effect size w = 0.04; post-hoc power = 0.11). Compared to the LLM-based method, Regex processing was more efficient, completing the task 28 120 times faster (0.06 seconds vs 1687.20 seconds). Further analysis revealed that LLMs favored common classifications (particularly BI-RADS value of 2) while Regex more frequently returned "unclear" values. We also could confirm in our sample an already known laterality bias for breast cancer (BI-RADS 6) and detected a slight laterality skew for suspected breast cancer (BI-RADS 5) as well. Conclusion For structured, standardized data like BI-RADS, traditional NLP techniques seem to be superior, though future work should explore hybrid approaches combining Regex precision for standardized elements with LLM contextual understanding for more complex information extraction tasks.