DATE: domain-aware transformer ensembles for reliable extremism detection in social media

Extremist content spreads quickly on social media, so moderation needs classifiers that are accurate and whose confidence can be trusted. We study the Domain-Aware Transformer Ensemble (DATE): a BERTweet-base classifier (pre-trained on 850M English tweets), contextual-substitution and random-edit augmentation applied only inside each training fold, and a soft-voting ensemble of ten fold models. We evaluate on the Social Media Extremism Detection dataset (2,250 training posts, 749 labelled test posts) with a leakage-controlled protocol and report three kinds of evidence separately. (i) Cross-validation: single DATE models reach 0.822 accuracy (±0.027) on held-out folds. (ii) Held-out test posts: the ten-model ensemble reaches 0.843 accuracy (95% CI 0.814–0.868), macro-F1 0.744 and ROC-AUC 0.863, and its expected calibration error is 0.030 against 0.064 for single models (paired bootstrap difference \(-0.029\) , CI \(-0.046\) to \(-0.010\) ); over five training seeds, the ensemble is better calibrated than its members in every run (mean ECE 0.037 against 0.061). (iii) Competition: the submitted DATE ensemble scored 0.8507 on the private leaderboard of the Kaggle challenge ( \(\approx\) 375 posts), tied for first place with two other teams among 164 teams; this margin is within sampling noise. Controlled ablations repeated over five seeds show that augmentation, class weights and the cosine schedule do not improve accuracy at full data size, although augmentation helps when training data is scarce; BERTweet beats RoBERTa-base in all five seeds (1.9 points on the test posts), and a TF-IDF baseline reaches comparable test accuracy. The study is limited to one English dataset whose annotation details (number of annotators, agreement) are not reported, and the test set has a different class balance from the training set.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-16
DOI
https://doi.org/10.1038/s41598-026-80011-4
Primary Topic
Hate Speech and Cyberbullying Detection
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

DATE: domain-aware transformer ensembles for reliable extremism detection in social media

Muhammad Zeeshan Jhandir, Imran Ashraf, Hafiz Muhammad Raza Ur Rehman, Mónica Gracia Villar et al.
1 links
Scientific Reports
Hate Speech and Cyberbullying Detection
article

DATE: domain-aware transformer ensembles for reliable extremism detection in social media

Muhammad Zeeshan Jhandir, Imran Ashraf, Hafiz Muhammad Raza Ur Rehman, Mónica Gracia Villar, Muhammad Ibrahim Qasim, Helena Garay, Zulqarnain Ali
article en

Abstract

Extremist content spreads quickly on social media, so moderation needs classifiers that are accurate and whose confidence can be trusted. We study the Domain-Aware Transformer Ensemble (DATE): a BERTweet-base classifier (pre-trained on 850M English tweets), contextual-substitution and random-edit augmentation applied only inside each training fold, and a soft-voting ensemble of ten fold models. We evaluate on the Social Media Extremism Detection dataset (2,250 training posts, 749 labelled test posts) with a leakage-controlled protocol and report three kinds of evidence separately. (i) Cross-validation: single DATE models reach 0.822 accuracy (±0.027) on held-out folds. (ii) Held-out test posts: the ten-model ensemble reaches 0.843 accuracy (95% CI 0.814–0.868), macro-F1 0.744 and ROC-AUC 0.863, and its expected calibration error is 0.030 against 0.064 for single models (paired bootstrap difference \(-0.029\) , CI \(-0.046\) to \(-0.010\) ); over five training seeds, the ensemble is better calibrated than its members in every run (mean ECE 0.037 against 0.061). (iii) Competition: the submitted DATE ensemble scored 0.8507 on the private leaderboard of the Kaggle challenge ( \(\approx\) 375 posts), tied for first place with two other teams among 164 teams; this margin is within sampling noise. Controlled ablations repeated over five seeds show that augmentation, class weights and the cosine schedule do not improve accuracy at full data size, although augmentation helps when training data is scarce; BERTweet beats RoBERTa-base in all five seeds (1.9 points on the test posts), and a TF-IDF baseline reaches comparable test accuracy. The study is limited to one English dataset whose annotation details (number of annotators, agreement) are not reported, and the test set has a different class balance from the training set.

Scientific Reports
Bahauddin Zakariya University (PK), Islamia University of Bahawalpur (PK), Universidad Europea del Atlántico (ES), Universidade Internacional do Cuanza (AO), Universidad Internacional Iberoamericana (PR), Fundación Universitaria Internacional de Colombia (CO), Universidad Internacional Iberoamericana (MX), Universidad de la Romana (DO), Yeungnam University (KR)
Hate Speech and Cyberbullying Detection
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon