How calibration fails when prediction models cross hospitals: logistic regression versus tree-based models for septic shock mortality

Abstract Background Clinical prediction models developed at single institutions rarely undergo rigorous external validation. Whether model calibration, the agreement between predicted and observed event rates, generalizes across populations with substantially different baseline risk remains poorly characterized. Methods We developed prediction models in MIMIC-IV (n = 17,516) and externally validated them in eICU-CRD (n = 2,819) using a pragmatic septic shock definition (suspected infection, organ dysfunction, and vasopressor use). Logistic regression (LR), random forest (RF), and XGBoost were compared on native probabilities using 72 predictors. We assessed discrimination (AUC), calibration (Brier score), and calibratability with small local samples (intercept correction and Platt scaling). Results Internal AUC ranking did not predict external performance. LR ranked second internally (0.820) but achieved the highest external AUC (0.813 vs. XGBoost 0.800, RF 0.759). LR's calibration slope stayed near 1.0, and its external error was a pure location shift that 50 local patients sufficed to correct. XGBoost's raw Brier of 0.185 reflected a coincidental match between its class-weight-centered predictions and eICU's 42.4% mortality, and its uncorrected calibration was prevalence-dependent in simulation. RF's error was a compression of predicted risks that a single intercept could not repair; only full recalibration reduced its Brier score (0.211 to 0.196). Conclusions When local recalibration data are scarce, the structure of calibration failure should guide model selection. LR's failure is visible and cheap to repair, XGBoost's is prevalence-dependent and may be invisible without local data, and RF's requires full recalibration rather than a single intercept.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-12
DOI
https://doi.org/10.21203/rs.3.rs-11105693/v1
Primary Topic
Sepsis Diagnosis and Treatment
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

How calibration fails when prediction models cross hospitals: logistic regression versus tree-based models for septic shock mortality

Haijin Suo, Jingyi Ren, Honghong Duan, Junjie Li et al.
Research Square
Sepsis Diagnosis and Treatment
preprint

How calibration fails when prediction models cross hospitals: logistic regression versus tree-based models for septic shock mortality

Haijin Suo, Jingyi Ren, Honghong Duan, Junjie Li, Jing Yang, Zelong Yang, Ying Zhang
preprint en

Abstract

Abstract Background Clinical prediction models developed at single institutions rarely undergo rigorous external validation. Whether model calibration, the agreement between predicted and observed event rates, generalizes across populations with substantially different baseline risk remains poorly characterized. Methods We developed prediction models in MIMIC-IV (n = 17,516) and externally validated them in eICU-CRD (n = 2,819) using a pragmatic septic shock definition (suspected infection, organ dysfunction, and vasopressor use). Logistic regression (LR), random forest (RF), and XGBoost were compared on native probabilities using 72 predictors. We assessed discrimination (AUC), calibration (Brier score), and calibratability with small local samples (intercept correction and Platt scaling). Results Internal AUC ranking did not predict external performance. LR ranked second internally (0.820) but achieved the highest external AUC (0.813 vs. XGBoost 0.800, RF 0.759). LR's calibration slope stayed near 1.0, and its external error was a pure location shift that 50 local patients sufficed to correct. XGBoost's raw Brier of 0.185 reflected a coincidental match between its class-weight-centered predictions and eICU's 42.4% mortality, and its uncorrected calibration was prevalence-dependent in simulation. RF's error was a compression of predicted risks that a single intercept could not repair; only full recalibration reduced its Brier score (0.211 to 0.196). Conclusions When local recalibration data are scarce, the structure of calibration failure should guide model selection. LR's failure is visible and cheap to repair, XGBoost's is prevalence-dependent and may be invisible without local data, and RF's requires full recalibration rather than a single intercept.

Research Square
Xijing Hospital (CN), Air Force Medical University (CN)
Good health and well-being
Sepsis Diagnosis and Treatment
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon