A comparative expert evaluation of ChatGPT and Gemini on urolithiasis: clinical accuracy versus citation reliability

The rapid evolution of large language models (LLMs) presents opportunities and risks, including ‘hallucination’, for urological practice. This study compares the performance of ChatGPT 5.2 Pro and Gemini 3 Pro in answering urolithiasis queries, and evaluates the discrepancy between clinical accuracy and citation reliability. In this cross-sectional study conducted between January and February 2026, 65 clinical questions based on the 2025 European Association of Urology (EAU) guidelines were administered to both models. Two blinded, board-certified urologists independently evaluated the 130 generated responses. The outputs were scored on three separate dimensions. These were medical accuracy, guideline adherence and citation quality. Each dimension was scored on a 1–5 scale. Gemini 3 Pro demonstrated statistically higher scores compared to ChatGPT 5.2 Pro in terms of medical accuracy and guideline adherence (both p < 0.001). Gemini 3 Pro provided responses that were rated as clinically appropriate by experts at a rate of 98.5%, compared to 81.5% for GPT-5.2 Pro ( p = 0.001). However, both models had critically flawed citation reliability. At the overall response level, ChatGPT generated high-risk responses in 95.4% of cases, compared to 69.2% for Gemini ( p < 0.001). At the individual citation level, 45.8% of GPT-5.2 Pro’s citations and 24.8% of Gemini 3 Pro’s citations were entirely fabricated (score 1a). Meanwhile, 30.3% and 31.4% of citations, respectively, were real publications with incorrect or mismatched DOIs (score 1b). Notably, there was no statistical correlation between medical accuracy and citation quality (ChatGPT 5.2 Pro: rho = 0.033, p = 0.793; Gemini 3 Pro: rho=-0.195, p = 0.120). While Gemini 3 Pro provides highly accurate, guideline-adherent clinical information for urolithiasis, neither model currently serves as a reliable primary tool for academic literature retrieval. There is a stark discrepancy between the models’ high clinical accuracy and their critically poor citation reliability. Therefore, it is unsafe to rely on AI-generated medical content without strictly and independently verifying the underlying references for clinical practice.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-18
DOI
https://doi.org/10.1038/s41598-026-81085-w
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A comparative expert evaluation of ChatGPT and Gemini on urolithiasis: clinical accuracy versus citation reliability

Yasin Aktaş, Adem Tunçekin
6 links
Scientific Reports
Artificial Intelligence in Healthcare and Education
article

A comparative expert evaluation of ChatGPT and Gemini on urolithiasis: clinical accuracy versus citation reliability

Yasin Aktaş, Adem Tunçekin
article en

Abstract

The rapid evolution of large language models (LLMs) presents opportunities and risks, including ‘hallucination’, for urological practice. This study compares the performance of ChatGPT 5.2 Pro and Gemini 3 Pro in answering urolithiasis queries, and evaluates the discrepancy between clinical accuracy and citation reliability. In this cross-sectional study conducted between January and February 2026, 65 clinical questions based on the 2025 European Association of Urology (EAU) guidelines were administered to both models. Two blinded, board-certified urologists independently evaluated the 130 generated responses. The outputs were scored on three separate dimensions. These were medical accuracy, guideline adherence and citation quality. Each dimension was scored on a 1–5 scale. Gemini 3 Pro demonstrated statistically higher scores compared to ChatGPT 5.2 Pro in terms of medical accuracy and guideline adherence (both p < 0.001). Gemini 3 Pro provided responses that were rated as clinically appropriate by experts at a rate of 98.5%, compared to 81.5% for GPT-5.2 Pro ( p = 0.001). However, both models had critically flawed citation reliability. At the overall response level, ChatGPT generated high-risk responses in 95.4% of cases, compared to 69.2% for Gemini ( p < 0.001). At the individual citation level, 45.8% of GPT-5.2 Pro’s citations and 24.8% of Gemini 3 Pro’s citations were entirely fabricated (score 1a). Meanwhile, 30.3% and 31.4% of citations, respectively, were real publications with incorrect or mismatched DOIs (score 1b). Notably, there was no statistical correlation between medical accuracy and citation quality (ChatGPT 5.2 Pro: rho = 0.033, p = 0.793; Gemini 3 Pro: rho=-0.195, p = 0.120). While Gemini 3 Pro provides highly accurate, guideline-adherent clinical information for urolithiasis, neither model currently serves as a reliable primary tool for academic literature retrieval. There is a stark discrepancy between the models’ high clinical accuracy and their critically poor citation reliability. Therefore, it is unsafe to rely on AI-generated medical content without strictly and independently verifying the underlying references for clinical practice.

Scientific Reports
Usak University (TR)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon