Comparative analysis of large language models and traditional diagnostic decision support systems for rare rheumatic disease identification

<h2>ABSTRACT</h2><h3>Objectives</h3> Rare rheumatic diseases are challenging to diagnose due to their complex and often atypical presentations. This study compared traditional diagnostic decision support systems (DDSSs) and large language models (LLMs) for rare rheumatic disease identification. <h3>Methods</h3> Comparison was based on 60 rare disease vignettes. Four general-purpose LLMs (Claude 3.5 Sonnet, ChatGPT-4o, Gemini 1.5 Pro, and Llama 3.3) and 3 traditional DDSSs (Symptoma, Ada, and Isabel DDx) were used to generate up to 5 disease suggestions per vignette based on anamnestic information. Identical diagnoses were scored with 2 points and plausible diagnoses with 1 point, contributing to a diagnostic score. Case completion time for each vignette was measured, and proportion of identical or plausible diagnoses were calculated. <h3>Results</h3> Overall diagnostic accuracy varied widely, with LLMs outperforming DDSSs. LLMs proposed an identical top diagnosis approximately 4 times more often than DDSSs (35.0% vs 8.9%; <i>P</i> < .001), and the average case diagnostic scores were 3.5 vs 2.1 for LLMs and DDSSs, respectively. LLMs were also more time-efficient than DDSSs with 20 vs 189 seconds per case. Claude 3.5 Sonnet achieved the highest overall diagnostic score (228) based on anamnestic information, followed by ChatGPT-4o (224), Llama 3.3 (200), Gemini 1.5 Pro (187), Symptoma (146), Ada (124), and Isabel DDx (116). <h3>Conclusions</h3> General-purpose LLMs outperformed the traditional DDSSs evaluated in this study, including both free and subscription-based medical products, in diagnostic accuracy and efficiency. Notably, open-source, locally hosted LLMs demonstrated promising performance, highlighting the potential of secure, on-premises LLMs for diagnostic decision support.