0348 Utility of Large Language Models for Extracting Sleep Apnea Specific Symptoms and Major Adverse Cardiovascular Events from Electronic Health Records

Abstract Introduction Electronic Health Record (EHR) clinical notes contain important symptom and event information that is often missing from structured fields. Automated extraction using large language models (LLMs) and supervised natural language processing (NLP) methods may support scalable clinical research. This study evaluates the utility of multiple methodological approaches, including prompt-based LLM extraction to supervised Bidirectional Encoder Representations from Transformers (BERT)-based classification and Named Entity Recognition (NER), for extraction of symptom and event information Methods Data from 215 suspected sleep apnea patients were assessed of whom 50 had documented sleep apnea symptoms, 145 had documented cardiovascular disease events and/or OSA diagnosis, and 20 had documented symptoms of cognitive impairment. All clinical notes were preprocessed using standard NLP pipelines. For sleep symptom and CVD event extraction, a hyperparameter-tuned BERT model was applied for both classification and NER tasks. For cognitive impairment symptom extraction, zero-shot prompting with Llama 3.1 405B generated structured outputs indicating symptom presence or absence. Model performance was evaluated using precision, recall, and F1 score, with F1>80% considered reliable. Results In the sleep symptom cohort, the BERT model achieved an average F1 score=0.86. In the CVD cohort, CAD was identified in 34 patients and OSA in 103 patients. The BERT model reliably classified CAD versus non-CAD (F1>0.80). False negatives were primarily due to missing documentation from external healthcare systems. Integration of ICD codes post hoc improved model specificity. LLM-based extraction successfully identified core cognitive impairment–related symptoms, with performance improving through targeted prompt engineering; however, hallucinations were observed with incomplete clinical documentation. Conclusion BERT models demonstrated reliable performance for extraction of OSA symptoms and CAD classification from unstructured EHR notes (F1> 0.80). LLM-based extraction of cognitive impairment–related symptoms showed moderate performance (F1 = 0.76) and was highly sensitive to prompt design and documentation quality. Overall, supervised NLP currently provides the most reliable approach for clinical event extraction from EHR notes, while LLMs show promise for more flexible symptom retrieval. Support (if any) R01HL171813, K25HL151912.