Evaluation of knowledge-augmented DeepSeek models on nephrology examination questions

Abstract Background Large language models (LLMs) demonstrate significant potential in medical applications. DeepSeek, as an open-source and customizable LLM, offers potential for nephrology applications through flexible deployment, potential cost reduction, and enhanced data privacy. However, its specific performance and applicability in the nephrology specialty remain to be systematically evaluated. Methods We evaluated six large language models (DeepSeek-R1, DeepSeek-V3, GPT-4, GPT-4o, Qwen-2.5, and Gemini-2.5) using 1 294 nephrology multiple-choice questions derived from the ASN NephSAP and KSAP educational programs. The DeepSeek-R1, DeepSeek-V3, GPT-4o, and Gemini-2.5 models were further enhanced through a retrieval-augmented generation (RAG) framework incorporating a nephrology comorbidity knowledge base (NCKB). The NCKB, reviewed by board-certified nephrologists, integrated authoritative sources including standard textbooks, clinical guidelines, and peer-reviewed literature. The NCKB included 50 347 structured knowledge units. Model accuracy, response time, and error types were analyzed using chi-square tests and visualized with bar and radar charts. Results A total of 1 294 nephrology exam questions were used for evaluation. The knowledge-enhanced DeepSeek models demonstrated significantly improved performance compared with their base versions (both P < 0.001), with DSR1 + NCKB and DSV3 + NCKB achieving accuracies of 81.22% and 79.13%, respectively. The performance difference between DSR1 + NCKB and DSV3 + NCKB did not reach statistical significance (P = 0.171). Knowledge gaps were the main cause of DSR1 + NCKB’s errors (52.26%), followed by reasoning and comprehension errors (20.16%). Conclusion Integrating an expert-curated NCKB significantly improves performance on standardized nephrology questions compared with baseline models. The magnitude of improvement varies across models, with DSR1 + NCKB achieving performance comparable to or approaching that of advanced proprietary models evaluated in this study.