Abstract 4144857: Large Language Models to Understand Reasons for Anticoagulation Nonprescription in Atrial Fibrillation

Introduction/Aims: Anticoagulation (AC) reduces the risk of stroke in atrial fibrillation (AF), yet nonprescription rates near 50%. Reasons for this care gap are not well understood. Large language models (LLMs) may be able to extract documented reasons for AC nonprescription from unstructured text in clinical notes. Methods: We created a cohort of all index patient encounters with a clinician-billed ICD10 diagnosis code for AF at Stanford Health Care from January 1, 2015 through December 31, 2023. Three trained reviewers annotated 10% of all AF- or AC-related note excerpts for whether the reason for AC nonprescription was documented and the reason for nonprescription with an overlap of 100 notes to assess annotator performance. Zero-shot prompts were designed for a proprietary LLM (GPT4) for the same tasks. Inter-annotator reliability (IRR) and model-annotator reliability (MAR) were assessed using Cohen’s kappa coefficient. Results: A total of 35,737 AF-related encounters were identified, of which 7,712 (21.6%) did not have an active AC prescription. These encounters contained 9,143 associated notes, from which 21,573 AF- and AC related excerpts were extracted. Ten percent of these (911 notes and 2,175 excerpts) were allocated for manual annotation. Reason for AC nonprescription was recorded in 497 (54.6%) notes, with antiplatelet use (N=172, 18.6%), perceived AC contraindication (N=134, 14.7%), and low burden of AF (N=131, 13.9%) being the most common. IRR across reviewers for identifying documented reasons was 0.776 (95% CI 0.716, 0.837), while MAR with GPT4 was 0.609, (95% CI 0.460, 0.758; P = 0.011). GPT4 achieved substantial agreement (MAR > 0.6) for identifying patient preference (MAR 0.763), antiplatelet use (MAR 0.676), and low stroke risk (mean MAR 0.891). There was moderate agreement for identifying perceived AC contraindication (MAR 0.589), fair agreement for low burden of AF (MAR 0.282), and poor agreement (MAR < 0.2) for identifying therapeutic inertia, existing AC use, and prior AF ablation. Conclusions: Across a contemporary AF cohort, LLMs categorized reasons for AC nonprescription including antiplatelet use as alternative agents, perceived AC contraindication, and low AF burden. This approach can be leveraged to understand and mitigate actionable barriers to guideline directed cardiovascular care.