A Multi-Chatbot Analysis: Strengths and Weaknesses in Neuroanatomy Learning

Background: The expanding interest in chatbots within the medical domain underscores the imperative for a comprehensive understanding of their capabilities and limitations, particularly in the context of anatomical education. Chatbots possess the potential to comprehend intricate anatomical concepts, deliver both advanced and contextually relevant information, and serve as a valuable resource for medical students and educators. This study aimed to evaluate the proficiency and constraints of chatbots in the domain of neuroanatomy. Methods: We developed 30 questions and administered them to ChatGPT-4, Google Gemini, Microsoft Copilot, and Perplexity.ai in their open versions. Questions were collaboratively constructed by the research team, selected through a semi-randomized process within the domain of neuroanatomy. Chatbots’ responses were evaluated in a blinded manner for validity and appropriateness, utilizing a 5-point Likert scale. Results: The highest observed performance among the evaluated chatbots was exhibited by ChatGPT-4 and Perplexity.ai, which achieved scores of 4.6 ± 0.5 and 4.5 ± 0.5, respectively. Microsoft Copilot (4.4 ± 0.5) and Google Gemini (4.1 ± 1.0) followed. The least successful performance was observed in the task of generating a neuroanatomical structure: only Microsoft Copilot attempted to fulfil the request, albeit with a dramatically flawed outcome. Conversely, Google Gemini and Perplexity.ai provided web links to anatomical illustrations. Conclusions: Despite technological advancements, AI models have not yet reached a level of sophistication sufficient to entirely supplant the role of educators or facilitators in a neuroanatomy course; however, they can serve as valuable adjunct tools for medical educators and students when utilized with careful consideration.