Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media: A Systematic Review and Empirical Evaluation
Mental disorders affect about 970 million people worldwide, yet few are ever screened in a clinic. Reddit posts contain people's own accounts of distress, making automated text classification an attractive, low-cost screening aid, but published results for competing transformer models use different datasets and protocols and cannot be compared. This study compared BERT, RoBERTa, MentalBERT, and DeBERTa-v3 under a single protocol to recommend a model for low-resource, multi-class screening. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was combined with a benchmark on Reddit posts labelled by subreddit into four classes: anxiety, depression, bipolar disorder and schizophrenia. Each model was fine-tuned with identical hyperparameters and class-weighted cross-entropy loss on a stratified sample of 20,001 posts on one NVIDIA T4 graphics processing unit (GPU), then tested on 3,001 held-out posts using macro-F1, accuracy and per-class precision and recall. Stability came from 30 bootstrap resamples of the test predictions, pairwise differences were tested with McNemar's test, and a decision matrix weighted macro-F1 (30%), domain pretraining relevance (25%), stability (15%), literature evidence (15%), efficiency (10%) and pretraining data overlap (5%). RoBERTa-base had the highest bootstrap macro-F1 (0.806), with DeBERTa-v3-base close behind (0.805). MentalBERT did not differ significantly from RoBERTa-base or BERT-base (p > .05). The matrix ranked MentalBERT first (8.07/10), but a sensitivity analysis showed that this depends on the judged criteria. Domain-adaptive pretraining gave no measurable advantage at this scale. Model choice should weigh accuracy, cost and domain fit together, and outputs should serve only as a triage signal.
Authors
- Casper Shikali Shivachi (ORCID: https://orcid.org/0000-0001-6479-5947)
- David Muturi
- Duncan Nyale (ORCID: https://orcid.org/0000-0003-0059-7655)
Institutions
- South Eastern Kenya University (KE)
- The Co-operative University of Kenya (KE)
Publication Details
- Journal
- International Journal of Professional Practice
- Published
- 2026-11-17
- DOI
- https://doi.org/10.71274/ijpp.v14i9.780
- Primary Topic
- Mental Health via Writing
- Type
- article
- Field-Weighted Citation Impact
- 0.00