Comment on ‘Chatbots in urology: accuracy, calibration, and comprehensibility; is <scp>DeepSeek</scp> taking over the throne?’

We read with great interest the study by Asker et al. [1], which systematically compared the performance of five mainstream large language models (LLMs), including ChatGPT-4o and DeepSeek-R1, in the field of urology. This work enriches research on the evaluation of emerging models and has substantial academic significance. Nevertheless, after careful review, we suggest that this study may have certain limitations in methodology and conclusion interpretation, which could benefit from further discussion to strengthen the study's findings. First, there are critical methodological limitations. The study assessed model performance utilising a mere 35 multiple-choice questions from the European Board of Urology in-service examination booklets, which cover seven urological subspecialties including andrology and LUTS [1]. The selected question bank was relatively limited, and its difficulty level and coverage were uneven. The number of questions across subfields (e.g., andrology, paediatric urology) was insufficient and imbalanced, potentially causing random fluctuations in model performance (e.g., Gemini-2.0 achieved only 58% accuracy in urological oncology) and making it difficult to reliably assess the models’ true proficiency in specialised subfields [2]. Second, the ‘understandability’ evaluation was subject to selection bias and had insufficient statistical power. The study included only 19 questions (with 95 associated explanatory passages) that ‘all models answered correctly’, rated by 25 participants predominantly comprising intern doctors (20 intern doctors and five urology residents with <1 year of experience) [1]. This selection approach inherently favours less difficult questions and uses a homogeneous rater sample, thereby failing to fully capture variations in how medical professionals with different experience levels understand the same text. Although anova revealed a statistically significant difference among models (P = 0.01), the effect size was small (η2 < 0.06), thereby limiting the practical relevance of the conclusion that ‘Claude 3.5 is more comprehensible’. Furthermore, the explanations from different models may have distinct stylistic features, which could theoretically undermine the rigour of the single-blind design. Meanwhile, for the assessment of readability, the study employed traditional metrics, such as Flesch Reading Ease and Gunning Fog [1]. While these metrics effectively assess the readability of general texts, their utility may be limited for texts dense with medical terminology – especially those involving complex clinical decision-making [3]. Last, but equally critical, like many studies on LLMs [2], this research lacks real-world clinical validation and has not been tested in clinical settings – thus, whether artificial intelligence-generated recommendations positively impact clinical decision-making remains unclear [4]. In summary, the study by Asker et al. offers a valuable preliminary exploration of performance comparisons among mainstream LLMs in urology. Notably, it highlights the competitiveness of the new model DeepSeek-R1 within this domain, thereby providing useful insights for related research directions. However, due to limitations including inadequate sample representativeness, poor suitability of evaluation metrics, and insufficient control over randomness, its core conclusions regarding the models’ accuracy, calibration, readability and understandability still require validation via a more rigorous framework. All authors declare no conflict of interest related to the present study. Kun Zhao and Zhongze Zhou wrote the main manuscript text. Zhilong Dong carried out the conceptualisation and supervision. All authors reviewed the manuscript. None. This article is linked to https://doi.org/10.1111/bju.16873.