Evaluation of GPT-4 Accuracy in the Interpretation of Medical Imaging: Potential Benefits, Limitations, and the Future

Introduction The implementation of artificial intelligence (AI) in radiology as a medical decision support system has the potential to enhance diagnostic accuracy and improve patient outcomes. This retrospective study aimed to evaluate the diagnostic capabilities of GPT-4o in interpreting radiological imaging, specifically X-ray, CT, and MRI images, across various organ systems and disease types. Methods A total of 377 cases were collected and presented to GPT-4o with a standardized prompt and no clinical context. The responses were assessed by three independent raters using a five-point rating system. Results X-ray imaging exhibited a 2.21 times higher chance, on average, of being interpreted accurately compared to CT scans (odds ratio (OR): 2.21; 95% confidence interval (CI): 1.33 - 3.69), while pelvic imaging had a 6.25 times lower chance, on average, of being interpreted accurately when compared to images of the abdomen (OR: 0.16; 95% CI: 0.02 - 0.56). Additionally, neoplastic conditions had a 2.7 times lower chance, on average, of being interpreted accurately compared to bleeding conditions (OR: 0.37; 95% CI: 0.16 - 0.84). Conclusion A bimodal distribution of median ratings highlights an overreliance on comparability to prior image encounters and emphasizes the need to develop a systematic approach to image analysis. Future research should prioritize eliminating hallucination, establishing standardized evaluation criteria, and exploring methods to integrate visual and text-based data in a balanced manner. Additionally, efforts should be directed towards enhancing dataset diversity to improve the model's overall accuracy and generalizability.