Transportability, dataset shift, and local recalibration of early ICU deterioration prediction across MIMIC-IV, eICU-CRD, and NWICU: a retrospective cross-system validation study

Abstract Background Clinical prediction models may retain discrimination after transport while producing unreliable absolute risks. We evaluated cross-system transportability of early ICU deterioration models and how local recalibration changed as local outcome-event information accumulated, together with the number of local patients required to accrue those events. Methods We developed penalized logistic regression and XGBoost models in MIMIC-IV using 17 predictors from ICU hours 0–24. Eligible adults were alive and still in the ICU at 24 hours and had not received invasive mechanical ventilation and did not have established tracheostomy during the first 24 hours. The primary outcome was ICU death or new invasive mechanical ventilation during hours 24–48; the 24–72-hour composite was secondary. Source models were evaluated without adaptation in eICU-CRD and NWICU. External cohorts were divided into patient-disjoint adaptation and fixed evaluation sets, and intercept-only (M1) or intercept-plus-slope (M2) recalibration was repeatedly estimated from samples targeting 20, 50, 100, or 200 local events. Results The MIMIC development cohort contained 31,054 ICU stays; eICU and NWICU contained 90,393 and 16,562 external stays. Primary event prevalences were 3.45%, 2.42%, and 2.75%. External discrimination did not collapse uniformly (AUROC 0.674–0.728), but calibration failures differed by environment and model. In eICU, calibration slopes were 0.893 for logistic regression and 1.003 for XGBoost; in NWICU, they were 1.234 and 1.054. Intercept-only recalibration was often effective for risk-level shift, whereas slope updating became more useful with increasing local event information when slope mismatch was present. At matched event targets, the 24–72-hour composite required 31.5%–32.6% fewer labeled patients in eICU and 33.9%–35.2% fewer in NWICU than the 24–48-hour outcome. Conclusions Transport failure was driven more consistently by environment-specific miscalibration than by loss of ranking performance. Recalibration benefit and stability depended on model, environment, failure mode, horizon, and local event information; the results do not support a universal event-count threshold.