Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media: A Systematic Review and Empirical Evaluation

Mental disorders affect about 970 million people worldwide, yet few are ever screened in a clinic. Reddit posts contain people's own accounts of distress, making automated text classification an attractive, low-cost screening aid, but published results for competing transformer models use different datasets and protocols and cannot be compared. This study compared BERT, RoBERTa, MentalBERT, and DeBERTa-v3 under a single protocol to recommend a model for low-resource, multi-class screening. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was combined with a benchmark on Reddit posts labelled by subreddit into four classes: anxiety, depression, bipolar disorder and schizophrenia. Each model was fine-tuned with identical hyperparameters and class-weighted cross-entropy loss on a stratified sample of 20,001 posts on one NVIDIA T4 graphics processing unit (GPU), then tested on 3,001 held-out posts using macro-F1, accuracy and per-class precision and recall. Stability came from 30 bootstrap resamples of the test predictions, pairwise differences were tested with McNemar's test, and a decision matrix weighted macro-F1 (30%), domain pretraining relevance (25%), stability (15%), literature evidence (15%), efficiency (10%) and pretraining data overlap (5%). RoBERTa-base had the highest bootstrap macro-F1 (0.806), with DeBERTa-v3-base close behind (0.805). MentalBERT did not differ significantly from RoBERTa-base or BERT-base (p > .05). The matrix ranked MentalBERT first (8.07/10), but a sensitivity analysis showed that this depends on the judged criteria. Domain-adaptive pretraining gave no measurable advantage at this scale. Model choice should weigh accuracy, cost and domain fit together, and outputs should serve only as a triage signal.

Authors

Institutions

Publication Details

Journal
International Journal of Professional Practice
Published
2026-11-17
DOI
https://doi.org/10.71274/ijpp.v14i9.780
Primary Topic
Mental Health via Writing
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media: A Systematic Review and Empirical Evaluation

Casper Shikali Shivachi, David Muturi, Duncan Nyale
International Journal of Professional Practice
Mental Health via Writing
article

Benchmarking Transformer-Based Language Models for Mental Health Text Classification on Social Media: A Systematic Review and Empirical Evaluation

Casper Shikali Shivachi, David Muturi, Duncan Nyale
article en

Abstract

Mental disorders affect about 970 million people worldwide, yet few are ever screened in a clinic. Reddit posts contain people's own accounts of distress, making automated text classification an attractive, low-cost screening aid, but published results for competing transformer models use different datasets and protocols and cannot be compared. This study compared BERT, RoBERTa, MentalBERT, and DeBERTa-v3 under a single protocol to recommend a model for low-resource, multi-class screening. A systematic review following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) was combined with a benchmark on Reddit posts labelled by subreddit into four classes: anxiety, depression, bipolar disorder and schizophrenia. Each model was fine-tuned with identical hyperparameters and class-weighted cross-entropy loss on a stratified sample of 20,001 posts on one NVIDIA T4 graphics processing unit (GPU), then tested on 3,001 held-out posts using macro-F1, accuracy and per-class precision and recall. Stability came from 30 bootstrap resamples of the test predictions, pairwise differences were tested with McNemar's test, and a decision matrix weighted macro-F1 (30%), domain pretraining relevance (25%), stability (15%), literature evidence (15%), efficiency (10%) and pretraining data overlap (5%). RoBERTa-base had the highest bootstrap macro-F1 (0.806), with DeBERTa-v3-base close behind (0.805). MentalBERT did not differ significantly from RoBERTa-base or BERT-base (p > .05). The matrix ranked MentalBERT first (8.07/10), but a sensitivity analysis showed that this depends on the judged criteria. Domain-adaptive pretraining gave no measurable advantage at this scale. Model choice should weigh accuracy, cost and domain fit together, and outputs should serve only as a triage signal.

International Journal of Professional PracticeVol. 14(9)
South Eastern Kenya University (KE), The Co-operative University of Kenya (KE)
Mental Health via Writing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon