Coverage collapse and deceptive calibration in EEG fatigue monitoring: a cross-dataset audit under within-session and cross-subject shift

Abstract Objective. Electroencephalography-based driver fatigue monitors are judged almost entirely on accuracy, yet a safety monitor is useful only if its stated confidence can be trusted when it is wrong. We audit whether it survives drift within a recording and transfer to a new person. Approach. Five model families, from a dense network to a gradient-boosted tree ensemble, run on three public corpora differing in channels, label instrument and elicitation. Calibration is measured pooled, per recording and per class, beside proper scoring rules. Seven conformal recalibration strategies are compared under one protocol, and we derive a ceiling on the coverage attainable by any rule thresholding a fixed calibration set. Main results. Pooled calibration error falls as accuracy falls, so the headline reliability metric improves as the model degrades. Aggregated per recording, the same predictions reverse it, exceeding the pooled estimate in all fifteen session-time bins of both continuously annotated corpora (sign test, p = 3.1 × 10⁻⁵); on the fatigued class the concealed error reaches 4.9 times the pooled value. Under transfer, class-balanced calibration error rises by 0.107 to 0.254 in all fifteen model-corpus cells, where coverage falls below its 0.90 target, to 0.533 at worst, with empty prediction sets in four. The ceiling held on all 32 subject streams and bound 20 below target, separating recoverable methods in advance. Rolling re-estimation under a finite-sample correction cleared it on 19 of those 20, held 0.886 to 0.909 in every cell, and cut between-subject coverage variability from 0.205 to 0.026 at constant update cost. Significance. Pooled calibration error and marginal coverage can both look healthy while a monitor is least reliable in the states that should raise an alarm. Three cheap diagnostics separate a calibrated monitor from one averaging failures of opposite sign, and one that covers from one that abstains.