Predicting acute intracranial vascular events in stroke alerts using large-language models and structured data

Acute stroke alerts are frequently activated for non-cerebrovascular conditions, creating false positives that strain resources and increase diagnostic uncertainty. We developed machine learning models integrating large-language models, structured electronic health record data, and clinical time-series features to predict acute intracranial vascular event (AIVE) at stroke alert activation. Using retrospective stroke alerts from the Mount Sinai Health System (2011–2021), we extracted demographics, comorbidities, medications, vital signs, laboratory results, and clinical notes documented before alert activation. AIVE was defined using radiographic gold standards as a composite of acute cerebrovascular and selected acute intracranial diagnoses. Notes were modeled using Clinical-Longformer, MxBAI embeddings with a feed-forward neural network, and BioWordVec embeddings with logistic regression; structured-data models used complementary hyperparameter-tuned workflows. Model outputs were combined in a late-fusion stacked ensemble and evaluated on internal and within-system held-out hospital test sets. Among 16,512 stroke alerts, 9,013 (55.6%) were due to AIVE. The multimodal model achieved an AUROC of 0.73 (95% CI, 0.71–0.75), positive predictive value of 0.68, sensitivity of 0.76, F1-score of 0.72, and specificity of 0.56 on internal testing, with similar held-out hospital performance. This approach may support post-activation stroke-alert risk stratification and reduce false-positive evaluations.