Interpretable Recalibration of a Sequential Organ Failure Assessment–Derived Score Using Machine Learning–Guided Optimization: Multicenter Cohort Study

Background: The Sequential Organ Failure Assessment (SOFA) score is central to risk stratification in intensive care units (ICUs), but its thresholds and 0- to 4-point increments were set by expert consensus in 1996 and never reestimated from outcome data. Machine learning models discriminate better while discarding the additive, point-based structure that makes SOFA auditable at the bedside, and methods for updating a legacy score rather than replacing it remain scarce. Objective: This study aimed to develop and internally validate a data-driven recalibration of a SOFA-derived score that learns thresholds and integer weights from outcome data while preserving an additive, interpretable point-based structure and to test whether it improves discrimination for ICU mortality over a prespecified SOFA-derived reference score. Methods: We conducted a retrospective multicenter cohort study using the eICU Collaborative Research Database (208 US hospitals, 2014-2015). We analyzed 14,848 adult first ICU admissions with complete day 1 SOFA-related measurements; 133,684 (90%) of 148,532 encounters with outcome data were excluded because of at least 1 missing determination. As vasoactive dose units could not be harmonized and urine output was largely unavailable, a prespecified SOFA-derived reference score served as the comparator. The framework couples supervised discretization (univariate decision trees and interval number chosen by a penalized validation criterion) with constrained integer-weight optimization under monotonicity, integrality, and scale-anchoring constraints, plus logistic calibration. Performance was estimated from pooled out-of-fold predictions of 10-fold nested cross-validation grouped by hospital, with bootstrap CIs; final thresholds and weights were refit on the full cohort and contributed no performance estimate. Results: ICU mortality was 17.03% (2529/14,848). The recalibrated score achieved an area under the receiver operating characteristic curve (AUROC) of 0.811 (95% CI 0.800-0.819) versus 0.751 (95% CI 0.740-0.761) for the reference score (paired difference 0.059, 95% CI 0.053-0.067), and an area under the precision-recall curve of 0.495 (95% CI 0.477-0.515) versus 0.393 (95% CI 0.377-0.411; paired difference 0.102, 95% CI 0.087-0.116). Improvement was consistent across all 10 hospital-grouped folds and length-of-stay strata. After unweighted logistic recalibration, global calibration was adequate; the Brier score was lower (0.112 vs 0.123), whereas other calibration-error measures were higher. Discrimination approached unconstrained XGBoost (Extreme Gradient Boosting; AUROC 0.818) and exceeded penalized logistic regression (AUROC 0.775) fitted on the same folds. The advantage persisted in all 1000 resampling replications and in a prespecified sensitivity analysis expanding the cohort to 61,469 encounters (median paired AUROC difference 0.037). Conclusions: Data-driven recalibration under explicit structural constraints can update the cutpoints and integer weights of an established clinical score without abandoning its additive, bedside-computable form and should transfer to other legacy instruments. The recalibrated score is a mortality risk model on its own 0-71 scale; an open-source calculator is provided, and external and prospective validation is required before clinical use.

Authors

Institutions

Publication Details

Journal
JMIR Medical Informatics
Published
2026-11-13
DOI
https://doi.org/10.2196/99361
Primary Topic
Sepsis Diagnosis and Treatment
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Interpretable Recalibration of a Sequential Organ Failure Assessment–Derived Score Using Machine Learning–Guided Optimization: Multicenter Cohort Study

Ancor Sanz‐García, Francisco Martín‐Rodriguez, Isabel de la Torre Díez, Agustín Mayo-Íscar et al.
4 links
JMIR Medical Informatics
Sepsis Diagnosis and Treatment
article

Interpretable Recalibration of a Sequential Organ Failure Assessment–Derived Score Using Machine Learning–Guided Optimization: Multicenter Cohort Study

Ancor Sanz‐García, Francisco Martín‐Rodriguez, Isabel de la Torre Díez, Agustín Mayo-Íscar, Juan Dueñas-Ruiz, Jaime Azpiazu, Yanelis Manzano Nieves, Anny María Hernández Cuello
article en

Abstract

Background: The Sequential Organ Failure Assessment (SOFA) score is central to risk stratification in intensive care units (ICUs), but its thresholds and 0- to 4-point increments were set by expert consensus in 1996 and never reestimated from outcome data. Machine learning models discriminate better while discarding the additive, point-based structure that makes SOFA auditable at the bedside, and methods for updating a legacy score rather than replacing it remain scarce. Objective: This study aimed to develop and internally validate a data-driven recalibration of a SOFA-derived score that learns thresholds and integer weights from outcome data while preserving an additive, interpretable point-based structure and to test whether it improves discrimination for ICU mortality over a prespecified SOFA-derived reference score. Methods: We conducted a retrospective multicenter cohort study using the eICU Collaborative Research Database (208 US hospitals, 2014-2015). We analyzed 14,848 adult first ICU admissions with complete day 1 SOFA-related measurements; 133,684 (90%) of 148,532 encounters with outcome data were excluded because of at least 1 missing determination. As vasoactive dose units could not be harmonized and urine output was largely unavailable, a prespecified SOFA-derived reference score served as the comparator. The framework couples supervised discretization (univariate decision trees and interval number chosen by a penalized validation criterion) with constrained integer-weight optimization under monotonicity, integrality, and scale-anchoring constraints, plus logistic calibration. Performance was estimated from pooled out-of-fold predictions of 10-fold nested cross-validation grouped by hospital, with bootstrap CIs; final thresholds and weights were refit on the full cohort and contributed no performance estimate. Results: ICU mortality was 17.03% (2529/14,848). The recalibrated score achieved an area under the receiver operating characteristic curve (AUROC) of 0.811 (95% CI 0.800-0.819) versus 0.751 (95% CI 0.740-0.761) for the reference score (paired difference 0.059, 95% CI 0.053-0.067), and an area under the precision-recall curve of 0.495 (95% CI 0.477-0.515) versus 0.393 (95% CI 0.377-0.411; paired difference 0.102, 95% CI 0.087-0.116). Improvement was consistent across all 10 hospital-grouped folds and length-of-stay strata. After unweighted logistic recalibration, global calibration was adequate; the Brier score was lower (0.112 vs 0.123), whereas other calibration-error measures were higher. Discrimination approached unconstrained XGBoost (Extreme Gradient Boosting; AUROC 0.818) and exceeded penalized logistic regression (AUROC 0.775) fitted on the same folds. The advantage persisted in all 1000 resampling replications and in a prespecified sensitivity analysis expanding the cohort to 61,469 encounters (median paired AUROC difference 0.037). Conclusions: Data-driven recalibration under explicit structural constraints can update the cutpoints and integer weights of an established clinical score without abandoning its additive, bedside-computable form and should transfer to other legacy instruments. The recalibrated score is a mortality risk model on its own 0-71 scale; an open-source calculator is provided, and external and prospective validation is required before clinical use.

JMIR Medical InformaticsVol. 14
Universidad de Valladolid (ES), Universidad Iberoamericana (MX), Universidad Europea del Atlántico (ES), Foundation for Transport and Energy Research and Development (ES), Universidade Internacional do Cuanza (AO), Universidad de la Romana (DO), Gerencia Regional de Salud de Castilla y León (Sacyl) (ES), University of Castilla-La Mancha (ES)
Good health and well-being
Sepsis Diagnosis and Treatment
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon