Evaluating the Role of GPT-4 and GPT-4o in the Detectability of Chest Radiography Reports Requiring Further Assessment

Purpose The aim of this study is to investigate the capability of generative pre-trained transformer 4 (GPT-4) and GPT-4o in identifying chest radiography reports requiring further assessment. Materials and methods This retrospective study included 100 cases from the National Institutes of Health chest radiography dataset, including 50 abnormal and 50 normal cases. A radiologist blinded to the study's purpose interpreted and reported the radiological findings for each case in English and separately determined the necessity for further assessment based on predefined criteria as referential standards. The radiology reports were then input into GPT-4 and GPT-4o models, accompanied by a prompt to identify cases requiring further assessment. This procedure was repeated five times in separate sessions for each model. Overall accuracy, sensitivity, and specificity of the necessity for further assessment were assessed using McNemar's test. Positive and negative predictive values were assessed using Fisher's exact test and Chi-square test, respectively. Results A total of 100 cases were included (mean age of 49.4 years ± 15.4 [standard deviation]; 56 women). Among them, 44 were judged by the radiologist to require further assessment. Across the five sessions, 19.6% and 35.8% of the cases were judged to require further assessment by GPT-4 and GPT-4o, respectively. The sensitivity, accuracy, and negative predictive value of GPT-4o (74.5%, 85.8%, and 82.6%, respectively) were all significantly higher than those of GPT-4 (44.5%, 75.6%, and 69.7%, respectively) (<i>p</i> < 0.001). The specificity and positive predictive value of GPT-4 (100% and 100%, respectively) were significantly higher than that of GPT-4o (94.6% and 91.6%, respectively) (<i>p</i> < 0.001). Conclusion GPT-4o showed acceptable performance in detecting chest radiography reports requiring further assessment.