DATE: domain-aware transformer ensembles for reliable extremism detection in social media
Extremist content spreads quickly on social media, so moderation needs classifiers that are accurate and whose confidence can be trusted. We study the Domain-Aware Transformer Ensemble (DATE): a BERTweet-base classifier (pre-trained on 850M English tweets), contextual-substitution and random-edit augmentation applied only inside each training fold, and a soft-voting ensemble of ten fold models. We evaluate on the Social Media Extremism Detection dataset (2,250 training posts, 749 labelled test posts) with a leakage-controlled protocol and report three kinds of evidence separately. (i) Cross-validation: single DATE models reach 0.822 accuracy (±0.027) on held-out folds. (ii) Held-out test posts: the ten-model ensemble reaches 0.843 accuracy (95% CI 0.814–0.868), macro-F1 0.744 and ROC-AUC 0.863, and its expected calibration error is 0.030 against 0.064 for single models (paired bootstrap difference \(-0.029\) , CI \(-0.046\) to \(-0.010\) ); over five training seeds, the ensemble is better calibrated than its members in every run (mean ECE 0.037 against 0.061). (iii) Competition: the submitted DATE ensemble scored 0.8507 on the private leaderboard of the Kaggle challenge ( \(\approx\) 375 posts), tied for first place with two other teams among 164 teams; this margin is within sampling noise. Controlled ablations repeated over five seeds show that augmentation, class weights and the cosine schedule do not improve accuracy at full data size, although augmentation helps when training data is scarce; BERTweet beats RoBERTa-base in all five seeds (1.9 points on the test posts), and a TF-IDF baseline reaches comparable test accuracy. The study is limited to one English dataset whose annotation details (number of annotators, agreement) are not reported, and the test set has a different class balance from the training set.
Authors
- Muhammad Zeeshan Jhandir (ORCID: https://orcid.org/0000-0002-5710-181X)
- Imran Ashraf (ORCID: https://orcid.org/0000-0002-8271-6496)
- Hafiz Muhammad Raza Ur Rehman (ORCID: https://orcid.org/0000-0003-2230-6927)
- Mónica Gracia Villar (ORCID: https://orcid.org/0000-0002-8547-9246)
- Muhammad Ibrahim Qasim
- Helena Garay
- Zulqarnain Ali
Institutions
- Bahauddin Zakariya University (PK)
- Islamia University of Bahawalpur (PK)
- Universidad Europea del Atlántico (ES)
- Universidade Internacional do Cuanza (AO)
- Universidad Internacional Iberoamericana (PR)
- Fundación Universitaria Internacional de Colombia (CO)
- Universidad Internacional Iberoamericana (MX)
- Universidad de la Romana (DO)
- Yeungnam University (KR)
Publication Details
- Journal
- Scientific Reports
- Published
- 2026-11-16
- DOI
- https://doi.org/10.1038/s41598-026-80011-4
- Primary Topic
- Hate Speech and Cyberbullying Detection
- Type
- article
- Field-Weighted Citation Impact
- 0.00