A comparative expert evaluation of ChatGPT and Gemini on urolithiasis: clinical accuracy versus citation reliability
The rapid evolution of large language models (LLMs) presents opportunities and risks, including ‘hallucination’, for urological practice. This study compares the performance of ChatGPT 5.2 Pro and Gemini 3 Pro in answering urolithiasis queries, and evaluates the discrepancy between clinical accuracy and citation reliability. In this cross-sectional study conducted between January and February 2026, 65 clinical questions based on the 2025 European Association of Urology (EAU) guidelines were administered to both models. Two blinded, board-certified urologists independently evaluated the 130 generated responses. The outputs were scored on three separate dimensions. These were medical accuracy, guideline adherence and citation quality. Each dimension was scored on a 1–5 scale. Gemini 3 Pro demonstrated statistically higher scores compared to ChatGPT 5.2 Pro in terms of medical accuracy and guideline adherence (both p < 0.001). Gemini 3 Pro provided responses that were rated as clinically appropriate by experts at a rate of 98.5%, compared to 81.5% for GPT-5.2 Pro ( p = 0.001). However, both models had critically flawed citation reliability. At the overall response level, ChatGPT generated high-risk responses in 95.4% of cases, compared to 69.2% for Gemini ( p < 0.001). At the individual citation level, 45.8% of GPT-5.2 Pro’s citations and 24.8% of Gemini 3 Pro’s citations were entirely fabricated (score 1a). Meanwhile, 30.3% and 31.4% of citations, respectively, were real publications with incorrect or mismatched DOIs (score 1b). Notably, there was no statistical correlation between medical accuracy and citation quality (ChatGPT 5.2 Pro: rho = 0.033, p = 0.793; Gemini 3 Pro: rho=-0.195, p = 0.120). While Gemini 3 Pro provides highly accurate, guideline-adherent clinical information for urolithiasis, neither model currently serves as a reliable primary tool for academic literature retrieval. There is a stark discrepancy between the models’ high clinical accuracy and their critically poor citation reliability. Therefore, it is unsafe to rely on AI-generated medical content without strictly and independently verifying the underlying references for clinical practice.
Authors
- Yasin Aktaş (ORCID: https://orcid.org/0000-0001-5255-3780)
- Adem Tunçekin (ORCID: https://orcid.org/0000-0001-9951-2556)
Institutions
- Usak University (TR)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-18
- DOI
- https://doi.org/10.1038/s41598-026-81085-w
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00