Benchmarking next-generation large language models (LLMS): Evaluation of patient-oriented prostate cancer guidance.

e17116 Background: Large language models (LLMs) are increasingly used by patients to seek medical information, including cancer-related guidance. However, the reliability and quality of LLM-generated responses for patient-oriented oncologic information remain insufficiently characterized. Early-stage prostate cancer (PCa) was chosen for evaluation due to its prevalence and preference-based treatment complexity that lead patients to seek outside information. We evaluated the performance of widely used LLMs in addressing patient-focused questions on early-stage PCa. Methods: We developed 47 questions based on National Comprehensive Cancer Network (NCCN) guidelines for patients on early stage prostate PCa, spanning seven clinical domains: general information, symptoms and risk factors, diagnostic testing, risk assessment, treatment options for early-stage disease, initial treatment for low-risk prostate cancer, and prostate-specific antigen (PSA) persistence and recurrence. Responses were generated using ChatGPT (GPT-5.2), DeepSeek (V3.2), and Gemini (2.5 Flash). Two board-certified oncologists independently evaluated each response for factual accuracy, clarity, coherence, relevance, and completeness using a 5-point Likert scale. Inter-rater reliability was assessed using Cohen’s kappa, and descriptive statistics were calculated for each LLM and domain. Results: Overall inter-rater agreement was low (κ = 0.05) across all domains. Despite this, all models achieved high absolute ratings, with mean scores ≥4.0 and median scores of 4–5 across most domains. ChatGPT demonstrated the highest performance across all quality domains, with mean ratings for clarity (4.68–4.89) and coherence (4.77–4.89) from both oncologists. Completeness scores were high for both ChatGPT (4.10–4.80) and Gemini (4.23–4.50). ChatGPT achieved the highest factual accuracy (4.10–4.90), followed by Gemini (4.18–4.75). Relevance ratings were similarly high for ChatGPT (4.34–4.90) and Gemini (4.45–4.52). Across clinical domains, ChatGPT and Gemini consistently achieved higher mean ratings than DeepSeek. ChatGPT showed the strongest performance for general information (4.0–5.0), symptoms and risk factors (4.73–5.0), risk assessment (4.75–5.0), and PSA persistence/recurrence (3.97–4.77). Gemini performed comparably across most domains, including diagnostic testing (4.49–4.73) and early-stage treatment (4.43–4.87). Conclusions: All evaluated LLMs provided generally high-quality responses to patient-oriented questions on early-stage PCa. However, poor inter-rater agreement highlights subjectivity in expert evaluation and underscores the need for standardized assessment frameworks. While ChatGPT and Gemini demonstrated greater consistency across domains, LLM-generated content should complement not replace clinician-guided patient education.