Utilizing large language models and natural language processing to classify ischemia status from cardiac stress tests in a large multicenter healthcare system

Four BERT large language models (LLMs) were fine-tuned, and a rules-based system was designed by training, validating, and testing on an annotated sample of 654 stress test reports from a multisite and multiyear dataset from the Veterans Health Administration (VHA). The LLM with the highest performance was a ClinicalBERT with precision, recall, and F1 of 86.4%, 100%, and 92.7%, respectively. The rules-based NLP system achieved similar results of 88.1%, 97.4%, and 92.5%, respectively. Stress test reports totaling 1,692,171 and representing 1,096,341 unique patients were classified using the rules-based system after ascertaining current technological limitations, and the system is presently operational for care quality evaluations. Utilizing NLP allows for accurate, high-throughput analysis of cardiac stress test text reports.