Specialized Large Language Model Outperforms Neurologists at Complex Diagnosis in Blinded Case-Based Evaluation

<b>Background/Objectives</b>: Artificial intelligence (AI), particularly large language models (LLMs), has demonstrated versatility in various applications but faces challenges in specialized domains like neurology. This study evaluates a specialized LLM's capability and trustworthiness in complex neurological diagnosis, comparing its performance to neurologists in simulated clinical settings. <b>Methods</b>: We deployed GPT-4 Turbo (OpenAI, San Francisco, CA, US) through Neura (Sciense, New York, NY, US), an AI infrastructure with a dual-database architecture integrating "long-term memory" and "short-term memory" components on a curated neurological corpus. Five representative clinical scenarios were presented to 13 neurologists and the AI system. Participants formulated differential diagnoses based on initial presentations, followed by definitive diagnoses after receiving conclusive clinical information. Two senior academic neurologists blindly evaluated all responses, while an independent investigator assessed the verifiability of AI-generated information. <b>Results</b>: AI achieved a significantly higher normalized score (86.17%) compared to neurologists (55.11%, <i>p</i> < 0.001). For differential diagnosis questions, AI scored 85% versus 46.15% for neurologists, and for final diagnosis, 88.24% versus 70.93%. AI obtained 15 maximum scores in its 20 evaluations and responded in under 30 s compared to neurologists' average of 9 min. All AI-provided references were classified as relevant with no hallucinatory content detected. <b>Conclusions</b>: A specialized LLM demonstrated superior diagnostic performance compared to practicing neurologists across complex clinical challenges. This indicates that appropriately harnessed LLMs with curated knowledge bases can achieve domain-specific relevance in complex clinical disciplines, suggesting potential for AI as a time-efficient asset in clinical practice.

Authors

Institutions

Publication Details

Journal
Brain Sciences
Published
2025-03-27
DOI
https://doi.org/10.3390/brainsci15040347
Citations
13
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
8.51
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Specialized Large Language Model Outperforms Neurologists at Complex Diagnosis in Blinded Case-Based Evaluation

Tanguy Madec, Jeanne Benoit, Nathan Torcida, Hugo Kermorvant et al.
13 citations
Brain Sciences
Artificial Intelligence in Healthcare and Education
8.51
article

Specialized Large Language Model Outperforms Neurologists at Complex Diagnosis in Blinded Case-Based Evaluation

Tanguy Madec, Jeanne Benoit, Nathan Torcida, Hugo Kermorvant, Adil Maarouf, Sami Barrit, Sofia Maldonado Slootjes, Stanislas Lagarde, Mejdeddine Al Barajraji, Alexis Robin, Sylvain Redon, Vincent Harlay, Aurélien Mazeraud, Sébastien Boulogne, Eva Diab, Salim El Hadwe, Bertil Delsaut, Alexandre Niset, Vito Tota, Sofiène Hadidane, Timothée Carette, Romain Carron, Joseph R. Madsen, Nicolas Massager, Thibault Carron
article en
13 citations

Abstract

<b>Background/Objectives</b>: Artificial intelligence (AI), particularly large language models (LLMs), has demonstrated versatility in various applications but faces challenges in specialized domains like neurology. This study evaluates a specialized LLM's capability and trustworthiness in complex neurological diagnosis, comparing its performance to neurologists in simulated clinical settings. <b>Methods</b>: We deployed GPT-4 Turbo (OpenAI, San Francisco, CA, US) through Neura (Sciense, New York, NY, US), an AI infrastructure with a dual-database architecture integrating "long-term memory" and "short-term memory" components on a curated neurological corpus. Five representative clinical scenarios were presented to 13 neurologists and the AI system. Participants formulated differential diagnoses based on initial presentations, followed by definitive diagnoses after receiving conclusive clinical information. Two senior academic neurologists blindly evaluated all responses, while an independent investigator assessed the verifiability of AI-generated information. <b>Results</b>: AI achieved a significantly higher normalized score (86.17%) compared to neurologists (55.11%, <i>p</i> < 0.001). For differential diagnosis questions, AI scored 85% versus 46.15% for neurologists, and for final diagnosis, 88.24% versus 70.93%. AI obtained 15 maximum scores in its 20 evaluations and responded in under 30 s compared to neurologists' average of 9 min. All AI-provided references were classified as relevant with no hallucinatory content detected. <b>Conclusions</b>: A specialized LLM demonstrated superior diagnostic performance compared to practicing neurologists across complex clinical challenges. This indicates that appropriately harnessed LLMs with curated knowledge bases can achieve domain-specific relevance in complex clinical disciplines, suggesting potential for AI as a time-efficient asset in clinical practice.

Brain SciencesVol. 15(4)
Université Claude Bernard Lyon 1 (FR), University of New Caledonia (NC), Boston Children's Hospital (US), Centre National de la Recherche Scientifique (FR), Cliniques Universitaires Saint-Luc (BE), Université Libre de Bruxelles (BE), Vrije Universiteit Brussel (BE), Harvard University (US), Inserm (FR), Université Côte d'Azur (FR), Université Paris Cité (FR), Aix-Marseille Université (FR), University of Cambridge (GB), Centre Hospitalier Universitaire de Grenoble (FR), Centre Hospitalier Universitaire Vaudois (CH), Centre Hospitalier Universitaire de Nice (FR), CHU Ambroise Paré (BE), Sorbonne Université (FR), Clinique Saint Pierre (BE), Centre Hospitalier Territorial de Nouvelle-Calédonie (NC), Centre Hospitalier Universitaire Amiens-Picardie (FR), Institut de Neurosciences des Systèmes (FR), Universitair Ziekenhuis Brussel (BE), Institut de Psychiatrie et Neurosciences de Paris (FR), Centre Hospitalier Universitaire de Tivoli (BE), Centre de Résonance Magnétique Biologique et Médicale (FR), FHU Neurovasc (FR), Hôpital de la Timone (FR), Chirurgie et extrémité céphalique, caractérisation morphologique et fonctionnelle (FR), UCLouvain (BE)
Openalex Percentile: Top 2%
Artificial Intelligence in Healthcare and Education
8.51
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.