LLMs Underperform on Classifying Anxiety and Depression Using Therapy Conversations: A First-Step Benchmark

Anxiety and depression are among the most prevalent mental health conditions worldwide. Early and accurate automated detection from naturalistic conversations (e.g., those recorded with a remote chatbot) could eventually improve screening and, in turn, access to timely care. As a first step towards this goal, we aim to evaluate the efficacy of both traditional machine learning and large language models (LLMs) in classifying anxiety and depression from psychotherapy sessions using labels derived from clinician-annotated session metadata reflecting the primary presenting psychiatric concerns. While psychotherapy transcripts do not reflect the real-world domain of remote naturalistic conversation, we conduct this analysis as an “easy” starting point towards the eventual goal of building generalizable, clinician-assistive models that can infer mental health status from unstructured, non-directive conversations captured in the home setting as part of a remote digital assessment process. LLM underperformance on a psychotherapy benchmark would indicate that LLMs are most likely not yet ready to advance towards mental health classifications in more complex and less structured contexts, such as from remote conversations with a chatbot or family member. To study whether LLMs can classify anxiety and depression from psychotherapy transcripts, we fine-tuned both established transformer models (BERT, RoBERTa, Longformer) and more recent large models (Mistral-7B), trained a Support Vector Machine using engineered features, and assessed prompting GPT chatbots. We observe that (1) all machine learning approaches perform poorly and (2) state-of-the-art models fail to improve multi-label classification performance relative to traditional machine learning methods, indicating the current limitations of using LLMs for classification of psychiatric diagnoses from unstructured patient text as of 2026.