Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study

Aim: To assess expert panel agreement with responses from three large language models, Claude, Gemini and ChatGPT, to questions based on guidelines for non-muscle-invasive bladder cancer (NMIBC), muscle-invasive bladder cancer (MIBC) and upper tract urothelial carcinoma (UTUC). Materials & methods: The 2025 European Association of Urology (EAU) guidelines were provided to Claude 3.5 Haiku, Gemini 2.5 Flash and GPT-4.1-mini. Figures and tables were converted into plain language text. Each model answered 25 questions per entity. Panels of 7–10 experts rated responses on a 9-point scale using a modified Delphi process with two rounds. Outcomes were consensus in agreement and median expert panel score ≥7. Results: For Claude, Gemini and ChatGPT, respectively, consensus in agreement was achieved for 88%, 100% and 96% of NMIBC responses (p = 0.097); 92%, 100% and 96% of MIBC responses (p = 0.223); and 56%, 100% and 96% of UTUC responses (p < 0.001). For UTUC, significantly fewer Claude responses achieved consensus in agreement than Gemini or ChatGPT responses. Across all questions, consensus in agreement was achieved for 59/75 Claude responses (78.7%), 75/75 Gemini responses (100%), and 72/75 ChatGPT responses (96%) (p < 0.001). Median scores ≥7 were achieved for 88%, 100% and 100% of NMIBC responses; 96%, 100% and 100% of MIBC responses; and 68%, 100% and 96% of UTUC responses. In UTUC, Claude had significantly fewer such responses than Gemini, but the difference from ChatGPT was not statistically significant. Overall, median scores ≥7 were achieved for 63/75 Claude responses (84%), 75/75 Gemini responses (100%) and 74/75 ChatGPT responses (98.7%) (p < 0.001). Conclusion: Under the study conditions, large language model responses frequently achieved expert panel agreement, particularly for Gemini and ChatGPT, supporting the potential of these models for guideline interpretation. Further validation is required to establish clinical utility and safety.

Authors

Institutions

Publication Details

Journal
Journal of Comparative Effectiveness Research
Published
2026-11-04
DOI
https://doi.org/10.57264/cer-2026-0079
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study

Eloy Vivas Vargas, Paula Pelechano Gómez, Elena Sevillano Fernandez, Alberto Perez-Lanzac et al.
Journal of Comparative Effectiveness Research
Artificial Intelligence in Healthcare and Education
article

Evaluation of large language models for guideline-based clinical questions in muscle-invasive and non-muscle-invasive bladder cancer and upper tract urothelial carcinoma: a multidisciplinary Delphi study

Eloy Vivas Vargas, Paula Pelechano Gómez, Elena Sevillano Fernandez, Alberto Perez-Lanzac, Antoni Vilaseca, Felipe Couñago, A. Capllonch Carrión, Richard Mast Vilaseca, Ángel Borque‐Fernando, Jesús María Fernández Gómez, J.L. Domínguez Escrig, L. Ladaria Sureda, Isabel Trias, Félix Guerrero‐Ramos, José Daniel Subiela, Julian Sanz, Luis Llanes González, Tomás Fernández Aparicio, María José Juan-Fita, Antonio López-Beltrán, Ana Loizaga Iriarte, Javier Martínez Benavides, Javier Puente Vazquez, Ana Cristina Plata Bello, Óscar Buisán Rueda, Felipe Villacampa Auba, Marta Trassierra Villa
article en

Abstract

Aim: To assess expert panel agreement with responses from three large language models, Claude, Gemini and ChatGPT, to questions based on guidelines for non-muscle-invasive bladder cancer (NMIBC), muscle-invasive bladder cancer (MIBC) and upper tract urothelial carcinoma (UTUC). Materials & methods: The 2025 European Association of Urology (EAU) guidelines were provided to Claude 3.5 Haiku, Gemini 2.5 Flash and GPT-4.1-mini. Figures and tables were converted into plain language text. Each model answered 25 questions per entity. Panels of 7–10 experts rated responses on a 9-point scale using a modified Delphi process with two rounds. Outcomes were consensus in agreement and median expert panel score ≥7. Results: For Claude, Gemini and ChatGPT, respectively, consensus in agreement was achieved for 88%, 100% and 96% of NMIBC responses (p = 0.097); 92%, 100% and 96% of MIBC responses (p = 0.223); and 56%, 100% and 96% of UTUC responses (p < 0.001). For UTUC, significantly fewer Claude responses achieved consensus in agreement than Gemini or ChatGPT responses. Across all questions, consensus in agreement was achieved for 59/75 Claude responses (78.7%), 75/75 Gemini responses (100%), and 72/75 ChatGPT responses (96%) (p < 0.001). Median scores ≥7 were achieved for 88%, 100% and 100% of NMIBC responses; 96%, 100% and 100% of MIBC responses; and 68%, 100% and 96% of UTUC responses. In UTUC, Claude had significantly fewer such responses than Gemini, but the difference from ChatGPT was not statistically significant. Overall, median scores ≥7 were achieved for 63/75 Claude responses (84%), 75/75 Gemini responses (100%) and 74/75 ChatGPT responses (98.7%) (p < 0.001). Conclusion: Under the study conditions, large language model responses frequently achieved expert panel agreement, particularly for Gemini and ChatGPT, supporting the potential of these models for guideline interpretation. Further validation is required to establish clinical utility and safety.

Journal of Comparative Effectiveness Research
Universidad de Oviedo (ES), Hospital Universitario de Canarias (ES), Bellvitge University Hospital (ES), Hospital Clínico San Carlos (ES), Hospital Costa del Sol (ES), Hospital General Universitario Morales Meseguer (ES), Hospital de Basurto (ES), Hospital Clínico Universitario de Valencia (ES), Hospital Universitario 12 De Octubre (ES), Fundación Instituto Valenciano de Oncología (ES), Hospital Universitario de Getafe (ES), Institut Català d'Oncologia (ES), Hospital Universitario Miguel Servet (ES), Hospital Clínic de Barcelona (ES), Vall d'Hebron Hospital Universitari (ES), Hospital Universitario Central de Asturias (ES), Clinica Universidad de Navarra (ES), Instituto de Investigación Sanitaria Aragón (ES), Hospital Universitario HM Sanchinarro (ES), Hospital Universitario Son Espases (ES), Hospital Universitario Ramón y Cajal (ES), University of Córdoba (ES), Universidad Europea de Madrid (ES)
Openalex Percentile: Top 22%
Artificial Intelligence in Healthcare and Education
DOIOpen Access PDFOpenAlex
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon