Operationalizing AI in Stroke Alerts: Balancing Sensitivity and Specificity in Predicting Acute Cerebrovascular Disease

Acute stroke alerts are frequently triggered by conditions unrelated to cerebrovascular disease, resulting in false positives that burden clinical teams and contribute to diagnostic ambiguity. At a large academic center, we developed ScanNER v2, a machine learning (ML) model based on large-language models (LLMs) and structured clinical data to predict the presence of acute cerebrovascular disease (ACD) in approximately 16,000 stroke alerts occurring over 10 years with an area under the receiver-operating curve and F1 score of 0.72 and overall positive predictive value of 0.68. In this perspective, we outline a practical framework for operationalizing this model within hospital-based stroke systems. We first describe our health-system experience developing and validating an AI-enabled pipeline, named "ScanNER 2," then take the point of view of two implementation angles (high sensitivity and high specificity), outlining the operational and clinical tradeoffs for each approach. We also highlight challenges related to implementation, clinical governance, workflow integration, and equity, emphasizing guardrails required for responsible deployment. As stroke centers increasingly adopt AI-assisted tools, this type of thought experiment is essential to ensure that such ML-based innovations effectively enhance the core mission of delivering timely, high-quality acute stroke care.