Machine Learning vs. Clinical Judgement for Extubation Failure: A Rigorous Re-evaluation

Background In 2022 we derived and validated the ABCDMed score to predict extubation failure. Machine learning models (MLM) for the same outcome now report areas under the receiver operating characteristic curve (AUC) of up to 0.98. We asked whether flexible algorithms, given the same patients may improve on a model built from clinical judgement. Methods. A secondary analysis of a original prospective cohort of 800 consecutive adults extubated on a ICU in Bogotá, Colombia, at 2,600 m above sea level was performed. Extubation failure prediction was compared with two MLM, random forest, gradient boosting, and with nine further strategies, under 10 times repeated stratified 5-fold cross validation in which imputation, scaling, resampling, variable selection and hyperparameter search were performed inside training folds only, and every arm received the same candidate variables. Artificial intelligence was used in two further roles: to appraise 14 published machine learning studies of this outcome against the same reporting items, and to produce an adversarial peer review of this manuscript, every claim of which was verified by re-analysis. Results. Extubation failure occurred in 71 patients (8.9%). The published points score reproduced exactly, separating four risk strata with observed failure rates of 4.4%, 7.7%, 11.6% and 23.0%, but the score distributions of patients who failed and succeeded overlap almost completely. Given the same 12 candidate variables, random forest reached 0.630 and gradient boosting 0.627 against 0.655 for ordinary logistic regression, a spread of 0.028 in a design that cannot detect less than 0.068. Given all 42 variables, tuning, assembling and class rebalancing, no strategy exceeded the original score; SMOTE oversampling left discrimination unchanged while inflating predicted risk by 75% and driving net benefit below the treat-no strategy. Among the 14 published studies, 12 displayed an explainability figure, 7 reported calibration, 4 had any external validation, 3 reported net benefit, 3 shared code and none used repeated or nested cross validation. Conclusions. Artificial intelligence excels at extracting patterns from existing datasets, but it cannot generate information that is not there. In this cohort, neither advanced MLM nor extensive algorithmic tuning outperformed a parsimonious model based on clinical judgment.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-19
DOI
https://doi.org/10.21203/rs.3.rs-11244575/v1
Primary Topic
Respiratory Support and Mechanisms
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Machine Learning vs. Clinical Judgement for Extubation Failure: A Rigorous Re-evaluation

Cristhian Rubio Ramos, Edith Elianna Rodriguez, David René Rodríguez Lima, Esther María Campo-Bautista
Research Square
Respiratory Support and Mechanisms
preprint

Machine Learning vs. Clinical Judgement for Extubation Failure: A Rigorous Re-evaluation

Cristhian Rubio Ramos, Edith Elianna Rodriguez, David René Rodríguez Lima, Esther María Campo-Bautista
preprint en

Abstract

Background In 2022 we derived and validated the ABCDMed score to predict extubation failure. Machine learning models (MLM) for the same outcome now report areas under the receiver operating characteristic curve (AUC) of up to 0.98. We asked whether flexible algorithms, given the same patients may improve on a model built from clinical judgement. Methods. A secondary analysis of a original prospective cohort of 800 consecutive adults extubated on a ICU in Bogotá, Colombia, at 2,600 m above sea level was performed. Extubation failure prediction was compared with two MLM, random forest, gradient boosting, and with nine further strategies, under 10 times repeated stratified 5-fold cross validation in which imputation, scaling, resampling, variable selection and hyperparameter search were performed inside training folds only, and every arm received the same candidate variables. Artificial intelligence was used in two further roles: to appraise 14 published machine learning studies of this outcome against the same reporting items, and to produce an adversarial peer review of this manuscript, every claim of which was verified by re-analysis. Results. Extubation failure occurred in 71 patients (8.9%). The published points score reproduced exactly, separating four risk strata with observed failure rates of 4.4%, 7.7%, 11.6% and 23.0%, but the score distributions of patients who failed and succeeded overlap almost completely. Given the same 12 candidate variables, random forest reached 0.630 and gradient boosting 0.627 against 0.655 for ordinary logistic regression, a spread of 0.028 in a design that cannot detect less than 0.068. Given all 42 variables, tuning, assembling and class rebalancing, no strategy exceeded the original score; SMOTE oversampling left discrimination unchanged while inflating predicted risk by 75% and driving net benefit below the treat-no strategy. Among the 14 published studies, 12 displayed an explainability figure, 7 reported calibration, 4 had any external validation, 3 reported net benefit, 3 shared code and none used repeated or nested cross validation. Conclusions. Artificial intelligence excels at extracting patterns from existing datasets, but it cannot generate information that is not there. In this cohort, neither advanced MLM nor extensive algorithmic tuning outperformed a parsimonious model based on clinical judgment.

Research Square
Hospital Universitario Mayor, Mederi (CO), Cardio, Inc. (Japan) (JP)
Respiratory Support and Mechanisms
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon