Large Language Models Predict Functional Outcomes After Acute Ischemic Stroke

BACKGROUND AND OBJECTIVES: Accurate prediction of functional outcomes after acute ischemic stroke can guide clinical decision-making and resource allocation. Prior work has relied predominantly on structured predictors and conventional machine learning, while the ability of large language models (LLMs) to infer modified Rankin Scale (mRS) scores from routine clinical admission notes is largely unexplored. METHODS: We evaluated encoder (Bidirectional Encoder Representations from Transformers [BERT], NYUTron) and generative (Llama-3.1-8B, MedGemma-4B) LLMs, in both frozen and fine-tuned settings, for discharge and 90-day mRS prediction. ModernBERT-base was evaluated as a long-context encoder sensitivity analysis. We retrospectively analyzed 9485 admission history and physical notes for discharge prediction and 1898 notes for 90-day prediction from the NYU Langone Get With The Guidelines–Stroke registry (2016-2025). The most recent 12 months were held out for testing. Performance was assessed for binary functional outcome (mRS 0-2 vs 3-6) and compared with structured, hybrid, and clinical risk score baselines. RESULTS: In frozen settings, the clinically pretrained NYUTron encoder outperformed all other models. After fine-tuning, Llama achieved the highest discrimination for discharge (area under the receiver operating characteristic curve [AUROC] 0.84, area under the precision–recall curve 0.81) and 90-day (AUROC 0.87, area under the precision–recall curve 0.84) prediction, while binary accuracy was similar among NYUTron, Llama, and MedGemma. A long-context ModernBERT-base sensitivity analysis did not outperform NYUTron or eliminate differences in exact mRS prediction. For 90-day prediction, fine-tuned Llama outperformed structured (Zhang et al, AUROC 0.82), hybrid (Sung et al, AUROC 0.82), and clinical risk score (Totaled Health Risks in Vascular Events, AUROC 0.80; Stroke Prognostication using Age and NIH Stroke Scale, AUROC 0.79) baselines (all P < .05). CONCLUSION: Fine-tuned LLMs can predict poststroke functional outcomes from admission notes alone and outperform the evaluated prognostic baselines. Task-specific adaptation altered relative model-class performance, with generative models achieving the strongest performance on several metrics; however, differences between model classes varied by metric and cannot be attributed to architecture alone. Our findings support the development of text-based prognostic tools that integrate into clinical workflows without manual data extraction.

Authors

Institutions

Publication Details

Journal
Neurosurgery
Published
2026-11-19
DOI
https://doi.org/10.1227/neu.0000000000004298
Citations
3
Primary Topic
Machine Learning in Healthcare
Type
article
Field-Weighted Citation Impact
2.90
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Large Language Models Predict Functional Outcomes After Acute Ischemic Stroke

Anton Alyakin, Jin Vivian Lee, Howard Antony Riina, Eric Karl Oermann et al.
3 citations
Neurosurgery
Machine Learning in Healthcare
2.90
article

Large Language Models Predict Functional Outcomes After Acute Ischemic Stroke

Anton Alyakin, Jin Vivian Lee, Howard Antony Riina, Eric Karl Oermann, Jennifer Ann Frontera, Yindalon Aphinyanaphongs, Krithik Vishwanath, Jinseok Lee, Eunice Yang, Annelene M. Schulze, Anjali Kapoor
article en
3 citations

Abstract

BACKGROUND AND OBJECTIVES: Accurate prediction of functional outcomes after acute ischemic stroke can guide clinical decision-making and resource allocation. Prior work has relied predominantly on structured predictors and conventional machine learning, while the ability of large language models (LLMs) to infer modified Rankin Scale (mRS) scores from routine clinical admission notes is largely unexplored. METHODS: We evaluated encoder (Bidirectional Encoder Representations from Transformers [BERT], NYUTron) and generative (Llama-3.1-8B, MedGemma-4B) LLMs, in both frozen and fine-tuned settings, for discharge and 90-day mRS prediction. ModernBERT-base was evaluated as a long-context encoder sensitivity analysis. We retrospectively analyzed 9485 admission history and physical notes for discharge prediction and 1898 notes for 90-day prediction from the NYU Langone Get With The Guidelines–Stroke registry (2016-2025). The most recent 12 months were held out for testing. Performance was assessed for binary functional outcome (mRS 0-2 vs 3-6) and compared with structured, hybrid, and clinical risk score baselines. RESULTS: In frozen settings, the clinically pretrained NYUTron encoder outperformed all other models. After fine-tuning, Llama achieved the highest discrimination for discharge (area under the receiver operating characteristic curve [AUROC] 0.84, area under the precision–recall curve 0.81) and 90-day (AUROC 0.87, area under the precision–recall curve 0.84) prediction, while binary accuracy was similar among NYUTron, Llama, and MedGemma. A long-context ModernBERT-base sensitivity analysis did not outperform NYUTron or eliminate differences in exact mRS prediction. For 90-day prediction, fine-tuned Llama outperformed structured (Zhang et al, AUROC 0.82), hybrid (Sung et al, AUROC 0.82), and clinical risk score (Totaled Health Risks in Vascular Events, AUROC 0.80; Stroke Prognostication using Age and NIH Stroke Scale, AUROC 0.79) baselines (all P < .05). CONCLUSION: Fine-tuned LLMs can predict poststroke functional outcomes from admission notes alone and outperform the evaluated prognostic baselines. Task-specific adaptation altered relative model-class performance, with generative models achieving the strongest performance on several metrics; however, differences between model classes varied by metric and cannot be attributed to architecture alone. Our findings support the development of text-based prognostic tools that integrate into clinical workflows without manual data extraction.

Neurosurgery
Washington University in St. Louis (US), Kyung Hee University (KR), NYU Langone Health (US), New York University (US), Columbia University (US), The University of Texas at Austin (US)
Good health and well-being
Openalex Percentile: Top 9%
Machine Learning in Healthcare
2.90
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon