Separating Probability Calibration from Selective Reliability in Transformer-based Stress Detection under Domain Shift

Probability calibration does not establish selective reliability in stress detection.This study jointly audits calibration, selective prediction, training variability, domain shift, and semantic stability on Dreaddit, controlling data leakage and accounting for post clustering.BERT, RoBERTa, and DeBERTa are evaluated across five paired training seeds using temperature scaling and label smoothing of strength 0.10, yielding 30 fits and 45 evaluation conditions.Both interventions reduce Brier score, negative log likelihood, and expected calibration error in every backbone and seed pair.Temperature scaling preserves F1 and area under the risk coverage curve in all 15 pairs.Label smoothing increases this area for RoBERTa across all seeds, while BERT and DeBERTa vary in direction.Archived domain holdout and topic stability analyses provide diagnostics conditional on fitted models.This integrated audit distinguishes findings consistent across retraining from conclusions conditional on fitted models, separating training variability from clustered test uncertainty.Generalization remains limited to Dreaddit.

Authors

Institutions

Publication Details

Journal
International journal of intelligent engineering and systems
Published
2026-11-17
DOI
https://doi.org/10.22266/ijies2026.1231.41
Primary Topic
Mental Health via Writing
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Separating Probability Calibration from Selective Reliability in Transformer-based Stress Detection under Domain Shift

Ema Utami, I Made Artha Agastya, Kusrini, Rizaldi
1 links
International journal of intelligent engineering and systems
Mental Health via Writing
article

Separating Probability Calibration from Selective Reliability in Transformer-based Stress Detection under Domain Shift

Ema Utami, I Made Artha Agastya, Kusrini, Rizaldi
article en

Abstract

Probability calibration does not establish selective reliability in stress detection.This study jointly audits calibration, selective prediction, training variability, domain shift, and semantic stability on Dreaddit, controlling data leakage and accounting for post clustering.BERT, RoBERTa, and DeBERTa are evaluated across five paired training seeds using temperature scaling and label smoothing of strength 0.10, yielding 30 fits and 45 evaluation conditions.Both interventions reduce Brier score, negative log likelihood, and expected calibration error in every backbone and seed pair.Temperature scaling preserves F1 and area under the risk coverage curve in all 15 pairs.Label smoothing increases this area for RoBERTa across all seeds, while BERT and DeBERTa vary in direction.Archived domain holdout and topic stability analyses provide diagnostics conditional on fitted models.This integrated audit distinguishes findings consistent across retraining from conclusions conditional on fitted models, separating training variability from clustered test uncertainty.Generalization remains limited to Dreaddit.

International journal of intelligent engineering and systemsVol. 19(12)
Universitas Amikom Yogyakarta (ID)
Mental Health via Writing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon