Decoding Complications: Large Language Model-Based Classification of Free-Text Entries in Clinical Stroke Registries

Clinical registries capture adverse events (AEs) in free-text fields that are rarely analyzed. We developed a reproducible large language model (LLM)-assisted workflow using 11,216 free-text entries from 18,069 patients in the German Stroke Registry; 9,146 informative entries were retained after filtering. Embedding-based clustering showed poor separation, whereas an LLM-guided taxonomy identified 89 leaf labels. On the model-assisted 1,000-entry development set used for model and prompt selection, GPT-5.4 reached F1 92.4% (95% confidence interval [CI] 91.4-93.4). On 300 held-out entries, weighted F1 against clinician consensus was 96.5% (94.6-98.1) for frozen GPT-5.4 labels and 90.5% (87.5-93.5) for locally run Qwen 3.5 27B. For AE concepts with a comparable structured field, free text documented 1,345 additional instances (+19.6% over 6,867 structured instances); it also documented 7,546 instances of AE concepts without a structured equivalent, most often pneumonia, urinary tract infection and delirium. Beyond baseline covariates and the structured AE count, integrated complication concepts increased the area under the curve for prediction of good functional outcome by 0.025 (95% CI 0.021-0.028) and for death by 0.018 (0.014-0.021); gains were minimal after including the discharge National Institutes of Health Stroke Scale (NIHSS) score. On 1,979 public coronary-trial fragments from the Aggregate Analysis of ClinicalTrials.gov (AACT), the software ran end to end, demonstrating its technical applicability to a second clinical text corpus. In conclusion, this workflow enables structured analysis of clinical registry free text and complements structured adverse-event documentation.

Authors

Institutions

Publication Details

Journal
Journal of Medical Systems
Published
2026-11-16
DOI
https://doi.org/10.1007/s10916-026-02486-y
Primary Topic
Machine Learning in Healthcare
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Decoding Complications: Large Language Model-Based Classification of Free-Text Entries in Clinical Stroke Registries

Taraneh Ebrahimi, Christian Thielscher, Gabor C. Petzold, Felix Jürgen Bode et al.
6 links
Journal of Medical Systems
Machine Learning in Healthcare
article

Decoding Complications: Large Language Model-Based Classification of Free-Text Entries in Clinical Stroke Registries

Taraneh Ebrahimi, Christian Thielscher, Gabor C. Petzold, Felix Jürgen Bode, Lara Klehr, Louisa Nitsch, Sebastian Stösser, Hannah Asperger, Omid Shirvani Samani, Julius Nicolai Meißner, Julia Layer, Niklas Michael von Danwitz, Johannes Weller, Sach Mukherjee, Franziska Dorn, on behalf of the GSR-ET Investigators
article en

Abstract

Clinical registries capture adverse events (AEs) in free-text fields that are rarely analyzed. We developed a reproducible large language model (LLM)-assisted workflow using 11,216 free-text entries from 18,069 patients in the German Stroke Registry; 9,146 informative entries were retained after filtering. Embedding-based clustering showed poor separation, whereas an LLM-guided taxonomy identified 89 leaf labels. On the model-assisted 1,000-entry development set used for model and prompt selection, GPT-5.4 reached F1 92.4% (95% confidence interval [CI] 91.4-93.4). On 300 held-out entries, weighted F1 against clinician consensus was 96.5% (94.6-98.1) for frozen GPT-5.4 labels and 90.5% (87.5-93.5) for locally run Qwen 3.5 27B. For AE concepts with a comparable structured field, free text documented 1,345 additional instances (+19.6% over 6,867 structured instances); it also documented 7,546 instances of AE concepts without a structured equivalent, most often pneumonia, urinary tract infection and delirium. Beyond baseline covariates and the structured AE count, integrated complication concepts increased the area under the curve for prediction of good functional outcome by 0.025 (95% CI 0.021-0.028) and for death by 0.018 (0.014-0.021); gains were minimal after including the discharge National Institutes of Health Stroke Scale (NIHSS) score. On 1,979 public coronary-trial fragments from the Aggregate Analysis of ClinicalTrials.gov (AACT), the software ran end to end, demonstrating its technical applicability to a second clinical text corpus. In conclusion, this workflow enables structured analysis of clinical registry free text and complements structured adverse-event documentation.

Journal of Medical SystemsVol. 50(1)
University Hospital Bonn (DE), German Center for Neurodegenerative Diseases (DE)
Good health and well-being
Machine Learning in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon