Often Correct, Not Reliably Correct: A Blinded Audit of GPT-5.2 as Guideline-Decision Support for a Rare Cancer (Penile Squamous Cell Carcinoma)

Objective: Large language models (LLMs) are increasingly used as guideline-decision support in oncology, but most validation has occurred in common cancers, where clinicians can usually recognize errors. The safety-critical setting is the opposite — rare cancers, where local expertise is limited, and errors may go unrecognized. Whether common-cancer concordance reflects the reliability needed there is unclear. Methods: We benchmarked GPT-5.2 against the EAU-ASCO 2025 Penile Cancer Guidelines using 100 guideline-derived questions across five domains and three zero-shot prompting strategies, yielding 300 responses. Two experienced urologists, blinded to prompt identity, scored concordance on a 3-point scale and classified errors as hallucination, omission, temporal, or reasoning. The primary analysis was a mixed-effects ordinal logistic regression with question as a random intercept; a cluster-robust logistic regression served as sensitivity. Reporting followed TRIPOD-LLM. Results: GPT-5.2 was fully concordant in 236 of 300 responses (78.7%; 95% confidence interval, 73.7–82.9; weighted κ=0.914). Prompt strategy did not improve performance (role-play +3 percentage points; 95% credible interval, −2 to +9). The domain effect was not robust once between-question clustering was modeled. Among 64 non-concordant responses, omission predominated (47.7%) and reasoning errors followed (37.5%); in treatment, omission accounted for 68.0% of errors, including loss of guideline-specified detail. Four questions were uniformly discordant across all three prompts. Conclusion: Headline concordance overstated clinical reliability. The clinically informative finding was not the overall accuracy but the structure of failure — omission of decision-critical treatment detail and stable errors. Prompt engineering did not rescue performance. Current LLM output may assist orientation in rare-cancer decision pathways but is not ready as stand-alone guideline-decision support.