Zero-Shot PI-RADS Version 2.1 Scoring with ChatGPT-4 Turbo and Llama 3: Diagnostic Performance and Agreement with Abdominal Radiologists

This retrospective, single-center study aimed to assess the diagnostic performance and agreement of two large language models (LLMs), ChatGPT-4 Turbo (OpenAI) and Llama 3 (Meta AI), in assigning Prostate Imaging Reporting and Data System (PI-RADS) scores to prostate MRI reports and to compare their performance with two abdominal radiologists. Structured prostate MRI reports (<i>n</i> = 500) obtained between January and December 2022, with original PI-RADS scores removed, were processed with LLMs using a standardized prompt to extract PI-RADS version 2.1 scores. Two abdominal radiologists independently assigned scores, with a third adjudicating discrepancies. Prostate biopsy results served as the reference standard for diagnostic performance assessment. There was high agreement between both models and radiologists: ChatGPT-4 Turbo and Llama 3 achieved 97.7% agreement (κ = 0.95), agreement between the LLM and radiologists ranged from 94.7% to 95.7% (κ = 0.89-0.91), and interradiologist agreement was 94.4% (κ = 0.88). ChatGPT-4 Turbo assigned significantly higher scores than radiologists (<i>P</i> < .005), while differences with Llama 3 were not statistically significant (<i>P</i> = .08). ChatGPT-4 Turbo and the original MRI reports achieved an area under the receiver operating characteristic curve (AUC) of 0.79 for predicting prostate cancer, and radiologists and Llama 3 achieved AUCs of 0.78 each. These results suggest that LLMs could improve prostate MRI reporting through accurate and consistent PI-RADS scoring. <b>Keywords:</b> Prostate, Oncology, Large Language Model <i>Supplemental material is available for this article.</i> © RSNA, 2025.