Evaluation of GPT-4o and GPT o1 pro in answering Turkish Neurosurgical Society proficiency board exam questions: A comparative study

Purpose: This study aims to evaluate and compare the answering capabilities and accuracy of GPT-4o and GPT o1 pro for a set of 98 questions, including image-based questions, from the 2024 neurosurgery board certification examination.Materials and methods: Answers to 98 questions, including image-based questions, were generated using GPT-4o and GPT o1 pro. We evaluated their performance through both one-shot and multiple-attempt methods, calculating accuracy and consistency rates. Comparisons were made between text-based and image-based questions to assess multimodal reasoning.Results: GPT o1 pro achieved a one-shot accuracy of 92.9%, while GPT-4o reached 71.4% (p<0.001). Over three attempts, GPT o1 pro maintained 91.8% accuracy compared to GPT-4o’s 72.4% (p<0.001). GPT o1 pro also showed higher consistency (91.8% vs 78.6%, p=0.009). For text-only questions, GPT o1 pro reached 96.3% one-shot accuracy, compared to GPT-4o’s 72.8%. For image-based questions, GPT o1 pro scored 76.5%, and GPT-4o scored 64.7%, though this difference was not statistically significant (p=0.68). These results suggest that GPT o1 pro outperforms GPT-4o, especially for text-based items, but both models need further improvement for image-based tasks.Conclusion: Both GPT o1 pro and GPT-4o successfully passed the 2024 Turkish Neurosurgical Society Board Certification Examination. GPT o1 pro showed notably higher accuracy and consistency than GPT-4o, suggesting a promising role for chain-of-thought reasoning in medical education.