Comparative Expert-Rated Response Quality of Large Language Models in Aneurysmal Subarachnoid Hemorrhage: A Multidimensional Evaluation
Background/Objectives: Large language models (LLMs) are increasingly used to obtain medical information, but the quality of their responses to complex, time-sensitive neurovascular questions requires disease-specific evaluation. We compared the expert-rated response quality of five publicly accessible LLM systems using standardized questions on aneurysmal subarachnoid hemorrhage (aSAH). Methods: Thirty standardized Turkish-language questions covering aneurysm treatment, vasospasm and related complications, and neurocritical care and follow-up (10 questions per domain) were submitted to ChatGPT, Gemini, Claude, DeepSeek, and Microsoft Copilot through public web interfaces in July 2026. Each question was submitted once per system in a separate chat session, generating 150 responses. Five senior neurosurgeons independently rated each response on a 1–10 scale for coverage, relevance, comprehensibility, clinical applicability, and overall quality. Inter-rater reliability was assessed using intraclass correlation coefficients (ICCs). Repeated-measures linear mixed models, Bonferroni-adjusted pairwise comparisons, and Friedman sensitivity analyses were performed. Results: The evaluation comprised 3750 expert score observations without missing data. Inter-rater reliability was good to excellent (ICC 0.835–0.911). AI system was significantly associated with all five evaluation dimensions (all p < 0.001). Adjusted overall scores were highest for ChatGPT (7.68), followed by Gemini (6.97), Claude (6.69), DeepSeek (5.92), and Copilot (4.75). Neither the clinical-topic main effect nor the model-by-topic interaction was statistically significant (all p > 0.05). Friedman analyses also identified between-model differences across all dimensions (all p < 0.001). Conclusions: Publicly accessible LLM systems differed significantly in expert-rated response quality when answering standardized Turkish-language aSAH questions. These differences were observed within the study’s specific question set and evaluation protocol and should not be interpreted as evidence of differences in real-world clinical outcomes. The absence of statistically significant model-by-topic interactions does not establish equivalent performance across clinical domains. Model-specific validation, assessment of clinically important errors, and continued expert oversight are needed before considering LLM-generated information for clinical decision support.
Authors
- Ahmet Tuğrul Şahin (ORCID: https://orcid.org/0000-0002-4855-696X)
- Yasin Taşkın (ORCID: https://orcid.org/0000-0002-9109-7826)
Institutions
- Tokat Gaziosmanpaşa Üniversitesi (TR)
Publication Details
- Journal
- Journal of Clinical Medicine
- Published
- 2026-11-13
- DOI
- https://doi.org/10.3390/jcm15228896
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00