Transportability, calibration, and alert burden of an hourly model for documented continuous vasopressor initiation: a retrospective three-database study

Background Clinical prediction models may retain discrimination across settings while losing probability calibration or a workable alert policy. We evaluated this distinction for hourly prediction of documented continuous vasopressor initiation. Methods We used MIMIC-IV v3.1, eICU-CRD v2.0, and SICdb v1.0.8. Untreated adult stays contributed hourly landmarks from ICU hour 6 through 24 while still under observation. The outcome was first initiation within six hours and before ICU/unit exit; exit without initiation remained an operational non-event. Histogram-based gradient boosting (HGB) and logistic regression were fitted and selected in patient-disjoint MIMIC-IV reference-period groups. Evaluation used a separate MIMIC-IV group and two external databases. Source observations were range-checked before predictor summarization; SICdb hourly aggregates were withheld until the source hour had ended. These retrospective timing rules did not verify availability in live clinical information systems. Results Evaluation included 165,204 landmarks from 9,534 MIMIC-IV reference-period test stays; 2,745,224 landmarks from 168,636 eICU-CRD stays; 58,756 landmarks from 3,997 SICdb stays. HGB AUROCs were 0.770, 0.849, 0.695, respectively. Across 100 identifier-disjoint local-update splits, median Brier skill relative to a calibration-subset constant forecast was −0.168, −0.005, 0.023 before updating and 0.019, 0.025, 0.028 after full logistic recalibration. With thresholds targeting five raw alerts per 100 calibration rows, event detection after six-hour suppression was 0.406, 0.510, 0.275; false emitted alerts were 1.64, 1.52, 1.51 per 100 evaluation rows. Thresholds targeting 80% raw landmark sensitivity detected 0.885, 0.873, 0.883 of eligible events after suppression, while 63.8%, 50.7%, 86.9% of event-free stays received at least one false alert. Conclusions Discrimination transferred unevenly, while calibration and alert performance remained setting-dependent. The evaluated model and alert policies are not ready for clinical deployment. Offline assessment against prespecified clinical criteria should precede any prospective silent evaluation.

Authors

Institutions

Publication Details

Journal
Frontiers in Medicine
Published
2026-11-13
DOI
https://doi.org/10.3389/fmed.2026.1993720
Primary Topic
Machine Learning in Healthcare
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Transportability, calibration, and alert burden of an hourly model for documented continuous vasopressor initiation: a retrospective three-database study

Xiaoye Xu, Pengqin Shu, Zhao Liu, Chao Chen et al.
3 links
Frontiers in Medicine
Machine Learning in Healthcare
article

Transportability, calibration, and alert burden of an hourly model for documented continuous vasopressor initiation: a retrospective three-database study

Xiaoye Xu, Pengqin Shu, Zhao Liu, Chao Chen, Cheng Bao, Fei Ying, Tielong Chen, Jing Sun
article en

Abstract

Background Clinical prediction models may retain discrimination across settings while losing probability calibration or a workable alert policy. We evaluated this distinction for hourly prediction of documented continuous vasopressor initiation. Methods We used MIMIC-IV v3.1, eICU-CRD v2.0, and SICdb v1.0.8. Untreated adult stays contributed hourly landmarks from ICU hour 6 through 24 while still under observation. The outcome was first initiation within six hours and before ICU/unit exit; exit without initiation remained an operational non-event. Histogram-based gradient boosting (HGB) and logistic regression were fitted and selected in patient-disjoint MIMIC-IV reference-period groups. Evaluation used a separate MIMIC-IV group and two external databases. Source observations were range-checked before predictor summarization; SICdb hourly aggregates were withheld until the source hour had ended. These retrospective timing rules did not verify availability in live clinical information systems. Results Evaluation included 165,204 landmarks from 9,534 MIMIC-IV reference-period test stays; 2,745,224 landmarks from 168,636 eICU-CRD stays; 58,756 landmarks from 3,997 SICdb stays. HGB AUROCs were 0.770, 0.849, 0.695, respectively. Across 100 identifier-disjoint local-update splits, median Brier skill relative to a calibration-subset constant forecast was −0.168, −0.005, 0.023 before updating and 0.019, 0.025, 0.028 after full logistic recalibration. With thresholds targeting five raw alerts per 100 calibration rows, event detection after six-hour suppression was 0.406, 0.510, 0.275; false emitted alerts were 1.64, 1.52, 1.51 per 100 evaluation rows. Thresholds targeting 80% raw landmark sensitivity detected 0.885, 0.873, 0.883 of eligible events after suppression, while 63.8%, 50.7%, 86.9% of event-free stays received at least one false alert. Conclusions Discrimination transferred unevenly, while calibration and alert performance remained setting-dependent. The evaluated model and alert policies are not ready for clinical deployment. Offline assessment against prespecified clinical criteria should precede any prospective silent evaluation.

Frontiers in MedicineVol. 13
Zhejiang Chinese Medical University (CN), Hangzhou Seventh Peoples Hospital (CN), Zhejiang University (CN)
Good health and well-being
Machine Learning in Healthcare
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon