Retrospective evaluation of transportability and uncertainty in ICU mortality prediction across MIMIC-IV and eICU

Abstract Background ICU prediction models must retain clinically meaningful performance across differences in patients, measurement practices and outcome recording. Discrimination alone cannot establish this reliability. We examined model transportability and prediction-set coverage under a common cross-database design. Methods In this retrospective study, adults still recorded in the index ICU stay at 24 hours were evaluated for subsequent recorded death before index ICU discharge. Four model families used six vital-sign concepts, common preprocessing and 27,437 training patients per source. Sixteen configurations per family underwent source-only screening followed by three-seed confirmation of two finalists. Mean source-validation AUROC determined selection before final test evaluation. Reciprocal testing, separate probability and conformal calibration, hospital-disjoint eICU partitions, overlap weighting and measurement thinning assessed complementary aspects of reliability. Results Cohorts comprised 54,376 MIMIC-IV and 102,581 eICU patients. Source validation selected a Transformer in both directions. Source-test and external AUROCs were 0.776 and 0.720 for MIMIC-IV development, and 0.767 and 0.738 for eICU development. MIMIC-IV-trained gradient boosting had higher external AUROC, 0.743. The selected models overestimated external mortality in one direction and underestimated it in the other. A domain classifier distinguished the databases with AUROC 0.957. Marginal prediction sets covered 88.9% and 91.1% of external outcomes, but only 1.4% and 0.0% of deaths. Class-conditional death coverage was 91.6% and 84.9%, with both labels included for 54.2% and 49.3% of patients. Separate target updating yielded death coverage of 88.9% and 91.4%, with two-label frequencies of 56.1% and 54.8%. Conclusions Common inputs and training budgets improved the interpretability of cross-database comparisons but did not establish measurement equivalence or clinical reliability. Source model selection, external probability accuracy and event-specific coverage require separate evaluation. Prediction-set coverage must be interpreted alongside informativeness and the target data required for updating.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11173088/v1
Primary Topic
Machine Learning in Healthcare
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Retrospective evaluation of transportability and uncertainty in ICU mortality prediction across MIMIC-IV and eICU

Tianyi Yu
Research Square
Machine Learning in Healthcare
preprint

Retrospective evaluation of transportability and uncertainty in ICU mortality prediction across MIMIC-IV and eICU

Tianyi Yu
preprint en

Abstract

Abstract Background ICU prediction models must retain clinically meaningful performance across differences in patients, measurement practices and outcome recording. Discrimination alone cannot establish this reliability. We examined model transportability and prediction-set coverage under a common cross-database design. Methods In this retrospective study, adults still recorded in the index ICU stay at 24 hours were evaluated for subsequent recorded death before index ICU discharge. Four model families used six vital-sign concepts, common preprocessing and 27,437 training patients per source. Sixteen configurations per family underwent source-only screening followed by three-seed confirmation of two finalists. Mean source-validation AUROC determined selection before final test evaluation. Reciprocal testing, separate probability and conformal calibration, hospital-disjoint eICU partitions, overlap weighting and measurement thinning assessed complementary aspects of reliability. Results Cohorts comprised 54,376 MIMIC-IV and 102,581 eICU patients. Source validation selected a Transformer in both directions. Source-test and external AUROCs were 0.776 and 0.720 for MIMIC-IV development, and 0.767 and 0.738 for eICU development. MIMIC-IV-trained gradient boosting had higher external AUROC, 0.743. The selected models overestimated external mortality in one direction and underestimated it in the other. A domain classifier distinguished the databases with AUROC 0.957. Marginal prediction sets covered 88.9% and 91.1% of external outcomes, but only 1.4% and 0.0% of deaths. Class-conditional death coverage was 91.6% and 84.9%, with both labels included for 54.2% and 49.3% of patients. Separate target updating yielded death coverage of 88.9% and 91.4%, with two-label frequencies of 56.1% and 54.8%. Conclusions Common inputs and training budgets improved the interpretability of cross-database comparisons but did not establish measurement equivalence or clinical reliability. Source model selection, external probability accuracy and event-specific coverage require separate evaluation. Prediction-set coverage must be interpreted alongside informativeness and the target data required for updating.

Research Square
King's College London (GB)
Good health and well-being
Machine Learning in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon