Diagnosing child and adolescent mental health conditions using generative AI: closed-source models are more accurate than open-source models

Timely and accurate diagnosis is required to reduce the risk of adverse developmental outcomes associated with untreated child mental health conditions. However, parents often face challenges in accessing diagnostic mental health services, leading them towards online information, including artificial intelligence (AI) enabled resources. As the popularity of AI enabled conversational agents (CAs) grows, their accuracy in diagnosing child and adolescent mental health conditions needs to be determined. We compared the diagnostic accuracy of two CAs, ChatGPT (closed-source) and Llama (open-source), in identifying common child and adolescent mental health conditions. Two clinical psychologists rated the diagnostic accuracy generated by each CA for 51 cases depicting child and adolescent mental health conditions. A Wilcoxon sum rank test revealed that ChatGPT was more accurate in diagnosing child mental health conditions than Llama ( W = 1741.5, p = .001, r = 0.30). Two-way ANOVA revealed that co-occurring conditions were diagnosed more accurately than intellectual developmental disorder ( MD = 0.94, p = .004, d = 1.39). These findings suggest that closed-source CAs present a lower risk of misdiagnosis than open-source CAs. The overall diagnostic accuracy of CAs is moderate and variable. Discussion between clinicians and families about the consultation of CAs should be encouraged.