Application of Large Language Models for Neurological Outcome Classification in Patients With Stroke: Development and Evaluation Study (Preprint)

<sec> <title>BACKGROUND</title> The modified Rankin scale (mRS) is an important metric in stroke research, often used as a primary outcome in clinical trials and observational studies. The mRS can be assessed retrospectively from electronic health records (EHR), though this process is labor-intensive and prone to inter-rater variability. Large language models (LLMs) have demonstrated potential in automating text classification </sec> <sec> <title>OBJECTIVE</title> We aim to create a fine-tuned LLM that can analyze EHR text and classify mRS scores for clinical and research applications. </sec> <sec> <title>METHODS</title> We performed a retrospective cohort study of patients admitted to a specialist stroke neurology service at a large academic hospital system between August 2020 and June 2023. Each patient’s medical record was reviewed at two time points: (1) hospital discharge and (2) approximately 90 days post-discharge. Two independent researchers assigned an mRS score at each time point. Two separate models were trained on EHR passages with corresponding mRS scores as labeled outcomes: (1) a multiclass model to classify all seven mRS scores and (2) a binary model to classify functional independence (mRS 0–2) versus non-independence (mRS 3–6). Four-fold cross-validation was conducted, using accuracy and Cohen's kappa as model performance metrics. </sec> <sec> <title>RESULTS</title> A total of 2,290 EHR passages with corresponding mRS scores were included in model training. The multiclass model—considering all seven scores of the mRS—attained an accuracy of 77% and a weighted Cohen's Kappa of 0.92. Class-specific accuracy was highest for mRS 4 (90%) and lowest for mRS 2 (28%). The binary model—considering only functional independence vs non-independence —attained an accuracy of 92% and Cohen's Kappa of 0.84. </sec> <sec> <title>CONCLUSIONS</title> Our findings demonstrate that LLMs can be successfully trained to determine mRS scores through EHR text analysis. With further advancements, fully automated LLMs could scale across large clinical datasets, enabling data-driven public health strategies and optimized resource allocation. </sec> <sec> <title>CLINICALTRIAL</title> stroke; modified Rankin scale; artificial intelligence; large language model; machine learning; electronic health records </sec>