ModernBert to LLaMA: a Benchmark Study of LLM Robustness for Depression Detection in Social Media Texts

Social media platforms generate vast amounts of textual data that offer critical insights into public mental health. However, effectively analyzing this data for early depression detection requires robust language models that can handle real-world variations. This study addresses the need for scalable and reliable AI-driven tools by exploring the effectiveness of various Large Language Models (LLMs) in classifying mental health status from social media text, aiming to identify architectures that are resilient to shifts in data volume, preprocessing, and domain. A comprehensive comparative evaluation of several prominent LLMs specifically LLaMA-1b, LLaMA-3b, ModernBert, Bert, GPT-2, and Pythia-1b was conducted. The models were systematically trained on subsets of the Sentiment140 dataset (100k and 50k rows) with and without text preprocessing to assess the impact of data volume and cleaning. Generalization was rigorously tested by evaluating all trained models on an independent, unseen, depression-focused dataset (DD-Kaggle). Performance was measured using accuracy, precision, recall, F1-score, and AUC. On in-distribution validation sets, a clear performance hierarchy emerged, with the larger LLaMA-3B model achieving a peak validation accuracy of 92.33% on 50k uncleaned dataset. However, on the unseen out-of-distribution test set, ModernBert achieved the highest accuracy, reaching a top score of 72.42%. This highlights that superior validation performance does not always guarantee the best real-world generalization. While increased training data volume consistently improved performance, the effect of preprocessing under the specific aggressive pipeline evaluated in this study was mixed and model-dependent, with no consistent improvement from cleaning. All models exhibited a significant performance drop from validation to the unseen test set, underscoring the challenge of domain shift. This study provides a critical benchmark for LLMs in mental health monitoring, demonstrating that evaluating models on independent test sets is essential for gauging true robustness. The findings indicate that while larger models excel on in-distribution data, architectural suitability, as seen with ModernBert, is a key factor for successful generalization. For developing reliable and effective mental health monitoring tools, rigorous testing that accounts for domain shift is paramount.