Abstract 4140236: Development and Accuracy of Natural Language Processing-based Expression Matching to Identify and Classify Cardiomyopathy from Cardiovascular Magnetic Resonance Reports

Background: Cardiovascular magnetic resonance (CMR) is the gold standard non-invasive assessment of cardiomyopathy (CM); however, interpretation and CM classification from reports by clinicians and researchers at scale can be unclear and time-consuming. Research Question: Can a large language model accurately classify CM from clinical CMR reports, to improve diagnostic accuracy? Aims: To establish a novel natural language processing (NLP) model employing expression matching for CM detection and differentiation, and to compare its effectiveness with established methods. Methods: In our study, an unsupervised rule-based NLP model was developed on 3561 consecutive text impressions from clinical CMR reports at Beth Israel Deaconess Medical Center and compared against physician adjudicated diagnosis (gold-standard labels) of 6 broad groups of CM ( Figure 1 ). A semi-supervised biomedical bidirectional encoder representation from transformers (BioBERT) model was fine-tuned (80% training/20% validation) using NLP labels to improve generalization. The models’ performance was compared with comprehensive keyword search and ChatGPT4. We performed external validation by testing 1202 clinical CMR text impressions from Brigham and Women’s Hospital. Results: The stepwise classification of our NLP model is shown in Figure 2 . NLP and NLP-based BioBERT outperformed other methods ( Table 1 ). In external validation, the mean AUC of NLP (0.92) and NLP-based BioBERT (0.93) had better performance compared to keyword search (0.71) and ChatGPT4 (0.73). Conclusion: Our rule-based NLP and NLP-based BioBERT models accurately identify CM from CMR reports, enabling applications in widespread CMR report comprehension and large dataset research of cardiomyopathies.