Evaluating AI in Psychiatry Board Exams: A Comparative Study of ChatGPT-4 and Google Gemini

Introduction: Artificial intelligence (AI) is revolutionizing medical education, with large language models (LLMs) such as ChatGPT-4 (OpenAI) and Google Gemini (Google AI) increasingly used as learning tools.This study examines ChatGPT-4 and Google Gemini's accuracy in answering board-level psychiatry examination questions and classifying question difficulty.Methods: This cross-sectional study evaluated ChatGPT-4 and Google Gemini using 993 validated board-style psychiatry questions from BoardVitals.AI models were tested using standardized prompts, and their responses were analyzed for accuracy and difficulty classification.Results: Both ChatGPT-4 and Google Gemini demonstrated high accuracy, significantly surpassing the peer benchmark of 75.95% (p<0.001).No statistically significant difference was found between the models in overall accuracy (ChatGPT-4: 90.4%, Google Gemini: 90.8%; p=0.658).Both models exhibited only fair agreement with BoardVitals' difficulty categorizations, with ChatGPT-4 (w=0.373)and Gemini (w=0.30)frequently underestimating difficult questions. Discussion and Conclusion: ChatGPT-4 and Google Gemini show high accuracy in answering psychiatry board-style questions, highlighting their potential as adjunctive tools in medical education.However, their limitations in higher-order reasoning and difficulty classification underscore the need for further refinement.Future research should explore AI integration into real-world clinical decision-making while ensuring human oversight to maintain reliability and ethical considerations.