Calibration and Missingness in External Confirmation of a Transparent Next-Day Extubation Model

Abstract Background Discrimination alone does not show whether a clinical prediction model transfers adequately. Calibration, predictive value, threshold behavior, and missingness can change the interpretation of favorable external discrimination. Methods We conducted a locked secondary analysis of the credentialed SCRIPT X2B8 version 1.0.0 dataset. A 10-feature respiratory comparator and a 37-feature logistic model were trained only on the supplied internal training cohort. Preprocessing and model specifications were frozen before confirmation. Thresholds selected on the internal validation cohort by maximizing Youden's J were applied unchanged to the internal test and institutional external cohorts. We assessed discrimination, calibration, prediction error, threshold metrics, patient-cluster uncertainty, and missingness-stratified performance. Results The internal test cohort contained 831 patient-day observations and 65 events. The full-model area under the receiver operating characteristic curve was 0.8553 versus 0.7916 for the comparator, a difference of 0.0636 (95% confidence interval 0.0236 to 0.1046). The external cohort contained 2,835 observations and 192 events. External area under the curve was 0.8490 versus 0.8183, a difference of 0.0307 (95% confidence interval 0.0030 to 0.0584). The full model had a worse external Brier score than the comparator, 0.0611 versus 0.0589, and a calibration slope of 0.675. At the frozen threshold, sensitivity was 0.885, specificity 0.683, positive predictive value 0.168, and negative predictive value 0.988. Full-model area under the curve declined from 0.8839 in the lowest external missingness quartile to 0.7681 in the highest. Conclusions The full model retained external discrimination and exceeded the respiratory comparator, but calibration, positive predictive value, and missingness findings limit readiness claims. These results support continued evaluation, not clinical use, deployment readiness, or patient benefit.