Separating Probability Calibration from Selective Reliability in Transformer-based Stress Detection under Domain Shift
Probability calibration does not establish selective reliability in stress detection.This study jointly audits calibration, selective prediction, training variability, domain shift, and semantic stability on Dreaddit, controlling data leakage and accounting for post clustering.BERT, RoBERTa, and DeBERTa are evaluated across five paired training seeds using temperature scaling and label smoothing of strength 0.10, yielding 30 fits and 45 evaluation conditions.Both interventions reduce Brier score, negative log likelihood, and expected calibration error in every backbone and seed pair.Temperature scaling preserves F1 and area under the risk coverage curve in all 15 pairs.Label smoothing increases this area for RoBERTa across all seeds, while BERT and DeBERTa vary in direction.Archived domain holdout and topic stability analyses provide diagnostics conditional on fitted models.This integrated audit distinguishes findings consistent across retraining from conclusions conditional on fitted models, separating training variability from clustered test uncertainty.Generalization remains limited to Dreaddit.
Authors
- Ema Utami (ORCID: https://orcid.org/0000-0002-8237-8693)
- I Made Artha Agastya (ORCID: https://orcid.org/0000-0002-8739-5767)
- Kusrini
- Rizaldi
Institutions
- Universitas Amikom Yogyakarta (ID)
Publication Details
- Journal
- International journal of intelligent engineering and systems
- Published
- 2026-11-17
- DOI
- https://doi.org/10.22266/ijies2026.1231.41
- Primary Topic
- Mental Health via Writing
- Type
- article
- Field-Weighted Citation Impact
- 0.00