Comparison between China-based DeepSeek and US-based major LLMs in answering social determinants of health questions in ophthalmology

Large language models (LLMs) have shown promise in ophthalmology. However, their performance in addressing social determinants of health (SDoH), critical drivers of eye care outcomes, remains unclear. This study evaluates 8 LLMs (DeepSeek-V3, DeepSeek-R1, ChatGPT o1, ChatGPT-4o, ChatGPT-4, ChatGPT-3.5, Gemini 1, and Gemini 1.5 Pro) on their ability to distinguish true from false statements related to SDoH and disparities in ophthalmology. A total of 238 referenced true statements were drawn from published literature and edited to create direct contradictions, i.e., false statements. The true and false statements were presented to the models, and each model was tested on its ability to identify the correct statement in each pair. Model accuracy ranged from 78.2 % to 92.0 %. Among the top three performers, two (DeepSeek-R1 and ChatGPT o1) were reasoning models. DeepSeek-V3 achieved the highest accuracy (not statistically superior than DeepSeek-R1 or ChatGPT o1) and significantly outperformed ChatGPT 4o, ChatGPT 4, ChatGPT 3.5, Gemini 1.5 Pro, and Gemini 1. Across all LLMs, performance was lowest on statements concerning pediatric eye disease (52.4 %, P = 0.016) and Asian race (70.3 %, P = 0.044). Gemini 1.5 Pro exhibited significantly lower accuracy on statements involving White race (P = 0.041) compared to other racial categories. These findings reveal knowledge gaps and potential biases in LLM understanding of SDoH in ophthalmology. Further investigation is needed to understand why such variations exist and to guide improvements in LLM performance on this important topic.