Comparison of Deepseek-V3, ChatGPT-4o, and online medical platforms in addressing chronic pancreatitis related questions

Chronic pancreatitis (CP) is a complex, progressive inflammatory disorder characterized by irreversible damage to the pancreatic parenchyma, leading to exocrine and endocrine dysfunction. Patients with CP frequently experience debilitating symptoms including chronic pain, malabsorption, and diabetes mellitus, significantly impacting their quality of life. A critical challenge in CP management is the limited accessibility to accurate, up-to-date medical information, which is essential for informed decision-making. In recent years, advanced large language models (LLMs) such as ChatGPT and DeepSeek have emerged as potential sources of medical knowledge, offering rapid responses to health-related queries. However, the reliability and clinical utility of these AI-driven platforms in the context of CP remain underexplored, particularly in comparison to established online medical resources. This study aimed to conduct a comprehensive evaluation of the performance of two leading AI models DeepSeek-V3 and ChatGPT-4o alongside conventional online medical platforms in addressing CP-related inquiries. The assessment focused on two key domains: (1) guideline-based knowledge derived from the 2020 American College of Gastroenterology (ACG) Clinical Guidelines, and (2) patient-specific concerns extracted from real-world clinical follow-up records. By comparing AI-generated responses to physician-curated information, this research sought to determine the potential role of LLMs in supporting CP patient education and clinical decision support. We conducted a two-phase evaluation comprising 21 guideline-derived questions and 40 patient-derived questions. In the first phase, two blinded gastroenterologists independently assessed responses from DeepSeek-V3 and ChatGPT-4o for accuracy, completeness, comprehensiveness, and safety. Their ratings were averaged for each question and model. In the second phase, responses from both models were compared with physician responses from online medical platforms. Forty patients rated responses from all three sources for empathy and satisfaction. Comparisons used paired Wilcoxon signed-rank tests or Friedman tests followed by paired Wilcoxon tests with Holm adjustment. Because participant-level ratings were unavailable for reanalysis, patient-reported outcomes were analyzed using matched question-level aggregated means. DeepSeek-V3 and ChatGPT-4o performed similarly across accuracy, completeness, comprehensiveness, and safety when answering guideline-based questions. For patient-derived questions, the three information sources showed comparable accuracy and safety. Although completeness differed in the omnibus analysis ( P = 0.015), none of the pairwise comparisons remained significant after Holm adjustment. Comprehensiveness also differed across sources ( P < 0.001), with DeepSeek-V3 receiving lower scores than both online medical platforms (Holm-adjusted P = 0.020) and ChatGPT-4o (Holm-adjusted P = 0.009); ChatGPT-4o and online medical platforms did not differ significantly (Holm-adjusted P = 0.499). Question-level mean ratings showed no significant differences among sources in perceived empathy or satisfaction. Across all sources, empathy was rated relatively favorably, whereas satisfaction remained comparatively modest. DeepSeek-V3 and ChatGPT-4o provided generally accurate and safe responses to chronic pancreatitis-related questions. However, DeepSeek-V3’s descriptively higher completeness did not translate into broader clinical coverage, as its comprehensiveness scores were lower than those of ChatGPT-4o and online medical platforms. LLMs may supplement patient education, but their responses should complement rather than replace professional clinical assessment, particularly when individualized interpretation and broader clinical context are required.