DeepSeek vs ChatGPT vs Claude: benchmarking large language models for clinical diagnosis using a novel ICD-10-CM-based evaluation framework

BACKGROUND: Large language models have shown potential for supporting clinical reasoning, but their performance in open-ended diagnostic tasks remains insufficiently evaluated using clinically grounded and reproducible criteria. Many prior studies have relied on multiple-choice medical benchmarks or subjective expert ratings, which may not fully reflect the uncertainty and diagnostic complexity of real-world clinical encounters. This study aimed to benchmark state-of-the-art LLMs for open-ended clinical diagnosis using a hierarchical ICD-10-CM-based evaluation framework. METHODS: We evaluated four LLMs: DeepSeek-V3, GPT-4o, GPT-4o mini, and Claude 3.5 Sonnet, on 50 real-world clinical cases from the publicly available MultiCaRe dataset. To reduce potential training-data leakage, only cases published after August 2024 were included. For each case, models generated three ranked differential diagnoses and corresponding ICD-10-CM codes. Diagnostic performance was assessed using a four-level hierarchical scoring framework based on ICD-10-CM structure, capturing both broad disease-category agreement and finer diagnostic specificity. Two independent runs were conducted per model-case pair to assess consistency. Descriptive statistics, pairwise one-tailed paired [Formula: see text]-tests, and correlation-based analyses were used to compare performance and robustness. RESULTS: DeepSeek-V3 achieved the highest mean diagnostic score (2.32 ± 1.53), followed by GPT-4o (2.23 ± 1.50), Claude 3.5 Sonnet (2.12 ± 1.53), and GPT-4o mini (2.10 ± 1.49). Binary adjusted diagnostic accuracy was 82% for DeepSeek-V3 and GPT-4o, 80% for GPT-4o mini, and 76% for Claude 3.5 Sonnet. Across all four models, performance was generally stronger for identifying correct disease categories than for making highly specific diagnoses. DeepSeek-V3 also showed the lowest run-to-run mean difference and the highest run-to-run correlation, indicating numerically greater stability. However, overall differences between models were modest. CONCLUSIONS: State-of-the-art LLMs demonstrated promising capability in open-ended clinical diagnosis, particularly for identifying clinically relevant disease categories. DeepSeek-V3 showed the strongest overall numerical performance in this benchmark, although differences across models were limited. The proposed ICD-10-CM-based hierarchical framework provides a reproducible and clinically meaningful approach for evaluating LLM diagnostic performance in open-ended settings.