The Potential of AI Chatbots as Diagnostic Tools in Mental Health: Evaluating Exercise Dependence Symptoms

Abstract This study aimed to evaluate the effectiveness of AI chatbots (Claude-3.5 Sonnet, ChatGPT-4o, and Gemini-1.5 Pro) in identifying exercise dependence symptoms using a hypothetical case study. To this end, three sport psychologists, each with a minimum of five years of experience, assessed the chatbots' performance in diagnostic assessment competency, implementation of diagnostic criteria, summary quality, terminology use, and overall effectiveness. The results indicated that while all chatbot models successfully identified major symptoms, their performance varied. Specifically, the Claude-3.5 Sonnet model demonstrated superior performance in specific areas, such as providing a clear case summary and using accurate terminology. However, all chatbots exhibited limitations in recognizing symptom severity and distinguishing between primary and secondary dependence. The sport psychologists expressed a willingness to use at least one AI chatbot model as an assistive tool in initial client assessments. These findings highlight the potential value of AI chatbots in mental health assessments. Future research should prioritize the optimization of algorithms and training data through expert collaboration and controlled real-world testing to improve the reliability and practical application of these tools.