Performance of large language models in urological decision support: a guideline-based comparative evaluation in urolithiasis

Large language models (LLMs) are increasingly investigated for their potential role in guideline-based clinical information support. However, their consistency with subspecialty guidelines, particularly in urolithiasis, remains underexplored. This study aimed to evaluate the performance of four large language models (LLMs); GPT-4, GPT-4-turbo, Claude, and Gemini in generating guideline-concordant responses to clinical questions related to urolithiasis. A total of 105 clinical questions were independently developed by the authors based on urolithiasis management principles. Each LLM generated responses in two separate sessions. Two experienced urologists evaluated the outputs for accuracy and concordance with guideline recommendations. Inter-rater agreement analysis demonstrated fair agreement between evaluators. Differences across models and guideline categories were assessed using appropriate statistical tests. All four LLMs demonstrated high guideline adherence, with mean total scores ranging from 86.5 ± 5.2 (Gemini) to 92.8 ± 3.1 (Claude). Claude achieved the highest correlation with expert ratings (r = 0.94, p < 0.01). There were no statistically significant differences across models or among the nine clinical categories (p > 0.05). Session-to-session repeatability was also high for all models, with intra-model correlation coefficients exceeding 0.90. LLMs, particularly Claude, can provide reliable, guideline-consistent answers to urolithiasis-related clinical queries. Their consistent performance across themes suggests utility as adjunctive informational tools for guideline-based urological education and support, although further validation in real-world clinical settings remains necessary.