Development and external validation of a machine learning model for predicting chronic critical illness in ICU patients with acute pancreatitis

BACKGROUND: To develop and validate a machine learning (ML) model to assess the risk of chronic critical illness (CCI) in intensive care unit (ICU) patients with acute pancreatitis (AP). METHODS: We utilised two large, publicly available ICU datasets, MIMIC-IV (v3.1) and the eICU Collaborative Research Database (v2.0), as the development cohort for model construction. A single-centre dataset from China (SZICU) was used for external validation. Three feature selection methods-stepwise regression, Least Absolute Shrinkage and Selection Operator (LASSO), and the Boruta algorithm-were employed. Three ML methods-logistic regression (LR), random forest (RF), and extreme gradient boosting (XGBoost)-were used for model development. Model performance was evaluated using the area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve (AUPRC), accuracy, F1 score, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and Brier score, in both internal and external validation. RESULTS: The incidences of CCI were 7.00%, 9.89%, and 20.09% in the training, internal validation, and external validation sets, respectively. Eight predictors of CCI were identified: calcium level, body temperature, vasopressor use, urine output, Glasgow Coma Scale score, albumin level, haemoglobin level, and a history of cerebrovascular disease. In the internal validation set, the RF model achieved an AUROC of 0.85 (0.77-0.91), an AUPRC of 0.53 (0.39-0.69), and a Brier score of 0.07 (0.05-0.09). In the external validation set, the RF model achieved an AUROC of 0.73 (0.64-0.81), an AUPRC of 0.42 (0.30-0.56), and a Brier score of 0.16 (0.12-0.20). Feature importance analysis revealed that calcium level, body temperature, vasopressor use, and urine output were the most influential predictors of CCI. CONCLUSIONS: We developed and validated an ML model using eight clinical variables to predict CCI risk in ICU patients with AP.