Battle of the Bots: Comparison of Artificial Intelligence Chatbots in a Literature Review of Randomized Controlled Trials in Urology

Background. Use of language-learning model-based artificial intelligence (AI) chatbots in literature reviews has not been explored. The present study seeks to evaluate and compare the quality of reference lists and scientific discussion provided by ChatGPT (Chat Generative Pre-Trained Transformer) (4o vs. 4) and Google Gemini. Methods. ChatGPT (4o-mini, paid vs. 4o, free) and Google Gemini were given prompts to identify, discuss results of, and provide related references to nine specific randomized controlled trials in urologic practice. Reference lists and associated discussion were graded using modified DISCERN criteria. Hallucination rates for unverifiable references were calculated; GPT-2 Output Detector and Sapling AI Detector were used to assess detection of AI-generated text. Mean ± SD scores were calculated and compared with independent-samples T-tests. Results. The target trial was identified in 11%, 22%, and 22% of prompts by ChatGPT 4o-mini, Google Gemini, and ChatGPT 4o, respectively. Hallucinations (out of ten) were high for ChatGPT4o and ChatGPT 4 (7.7 and 6.3, respectively); in comparison, Google Gemini was significantly less likely to hallucinate (0.17, p < 0.005). Modified DISCERN scores were poor for all three chatbots (range: 26.6 to 29.7). AI output was detected in 60-72% of responses via GPT-2 and Sapling AI. Conclusion. AI chatbots were unable to identify the target article in most instances. Only Google Gemini appropriately refused to provide related articles and/or answer open-ended prompts if the target article could not be identified. Detection of AI-generated text was high for all chatbots, and caution should be used when using these AI resources in scientific writing or literature review.