Putting ChatGPT to the Test: Accuracy and Error Patterns in Clinical and Physics-Based Radiation Oncology Questions

Objective Large language models (LLMs) are increasingly being used as revision tools. Radiation oncology encompasses a complex skill set, including clinical staging, trials, anatomy, radiobiology, physics, mathematical calculations, treatment planning, and quality assurance (QA). The accuracy, reasonability, and error types of two ChatGPT models were assessed across clinical and medical physics radiation oncology examination question banks. Methodology Two independent scoring studies were undertaken (n = 200). Cohort 1 assessed the accuracy of ChatGPT 5.5 across 100 single-best-answer clinical questions from the American College of Radiology (ACR) Radiation Oncology In-Training Examination (TXIT). Questions were selected at random using a computer-generated random sampling script (25 questions per year from the years 2021, 2024, 2025, and 2026). Cohort 2 assessed ChatGPT 5.6 (Luna build, v5.6) across 100 RAPHEX-style radiation oncology physics multiple-choice questions covering five domains of physics. Each question was asked once using a standardized zero-shot prompt requesting a single best answer with reasoning. Accuracy was scored against published answer keys. Response length, error types, and exploratory scoring of reasoning quality (0-2) were noted. Chi-square (χ²), Fisher's exact, Welch's t-test, and Mann-Whitney U tests were used to compare results (P = 0.05). Results ChatGPT 5.5 had an overall accuracy of 82.0% (82/100) (95% confidence interval (CI): 72.8%-89.0%). Accuracy was 100% for radiobiology, pediatrics, and renal/bladder oncology. Accuracy was lowest for gynecological oncology (female) at 60.0% and benign/palliative radiation oncology at 66.7%. Accuracy varied between years of examination (2021-2026) from 76.0% to 88.0% (χ² = 2.44, df = 3, P = 0.49). Incorrect responses had a greater mean length compared to correct responses (227.657.5 vs. 202.144.3) (Welch's t = 1.77, P = 0.09) (Mann-Whitney U = 551.5, P = 0.10). Error types were mainly guideline/trial/numerical-specific recall errors (5/18, 27.8%). ChatGPT 5.6 had an overall accuracy of 87.0% (87/100) (95% CI: 78.7%-93.2%). Accuracy was highest for brachytherapy (15/15, 100%) and imaging/planning (24/25, 96.0%). Accuracy was lowest for beam physics (21/27, 77.8%). Mean length of reasoning was 186.848.2 words (correct = 188.4, incorrect = 176.2) (Student's t = 0.81, P = 0.42). Error types were most commonly conceptual errors (5/13, 38%) and self-contradictory responses (4/13, 31%). Conclusions Both ChatGPT 5.5 and ChatGPT 5.6 had high agreement with the answer keys. However, error types differed. Clinical errors were mainly due to detailed guideline/trial recall and anatomical staging errors. Physics errors were often due to conceptual errors and mathematical contradictions. LLMs can be useful as revision tools; however, responses must be validated against authoritative literature.

Authors

Institutions

Publication Details

Journal
Cureus
Published
2026-11-18
DOI
https://doi.org/10.7759/cureus.119879
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Putting ChatGPT to the Test: Accuracy and Error Patterns in Clinical and Physics-Based Radiation Oncology Questions

Veronia Fahmy, James S Welsh, Jane Georgas
4 links
Cureus
Artificial Intelligence in Healthcare and Education
article

Putting ChatGPT to the Test: Accuracy and Error Patterns in Clinical and Physics-Based Radiation Oncology Questions

Veronia Fahmy, James S Welsh, Jane Georgas
article en

Abstract

Objective Large language models (LLMs) are increasingly being used as revision tools. Radiation oncology encompasses a complex skill set, including clinical staging, trials, anatomy, radiobiology, physics, mathematical calculations, treatment planning, and quality assurance (QA). The accuracy, reasonability, and error types of two ChatGPT models were assessed across clinical and medical physics radiation oncology examination question banks. Methodology Two independent scoring studies were undertaken (n = 200). Cohort 1 assessed the accuracy of ChatGPT 5.5 across 100 single-best-answer clinical questions from the American College of Radiology (ACR) Radiation Oncology In-Training Examination (TXIT). Questions were selected at random using a computer-generated random sampling script (25 questions per year from the years 2021, 2024, 2025, and 2026). Cohort 2 assessed ChatGPT 5.6 (Luna build, v5.6) across 100 RAPHEX-style radiation oncology physics multiple-choice questions covering five domains of physics. Each question was asked once using a standardized zero-shot prompt requesting a single best answer with reasoning. Accuracy was scored against published answer keys. Response length, error types, and exploratory scoring of reasoning quality (0-2) were noted. Chi-square (χ²), Fisher's exact, Welch's t-test, and Mann-Whitney U tests were used to compare results (P = 0.05). Results ChatGPT 5.5 had an overall accuracy of 82.0% (82/100) (95% confidence interval (CI): 72.8%-89.0%). Accuracy was 100% for radiobiology, pediatrics, and renal/bladder oncology. Accuracy was lowest for gynecological oncology (female) at 60.0% and benign/palliative radiation oncology at 66.7%. Accuracy varied between years of examination (2021-2026) from 76.0% to 88.0% (χ² = 2.44, df = 3, P = 0.49). Incorrect responses had a greater mean length compared to correct responses (227.657.5 vs. 202.144.3) (Welch's t = 1.77, P = 0.09) (Mann-Whitney U = 551.5, P = 0.10). Error types were mainly guideline/trial/numerical-specific recall errors (5/18, 27.8%). ChatGPT 5.6 had an overall accuracy of 87.0% (87/100) (95% CI: 78.7%-93.2%). Accuracy was highest for brachytherapy (15/15, 100%) and imaging/planning (24/25, 96.0%). Accuracy was lowest for beam physics (21/27, 77.8%). Mean length of reasoning was 186.848.2 words (correct = 188.4, incorrect = 176.2) (Student's t = 0.81, P = 0.42). Error types were most commonly conceptual errors (5/13, 38%) and self-contradictory responses (4/13, 31%). Conclusions Both ChatGPT 5.5 and ChatGPT 5.6 had high agreement with the answer keys. However, error types differed. Clinical errors were mainly due to detailed guideline/trial recall and anatomical staging errors. Physics errors were often due to conceptual errors and mathematical contradictions. LLMs can be useful as revision tools; however, responses must be validated against authoritative literature.

Cureus
Loyola University Medical Center (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon