Do Clinical Notes Improve Early Prediction of Mortality or Prolonged ICU Stay?

Studies that add clinical notes to structured electronic health record data report gains in early prediction of in-hospital death or prolonged ICU stay ranging from none to 0.064 in the area under the receiver operating characteristic curve (AUROC). Each study measures that gain against a different structured baseline. In 25,692 MIMIC-III admissions, we held the cohort, outcome and notes fixed and varied only the structured baseline. Against 21 structured predictors, notes increased the AUROC by 0.065; against 223, by 0.010 under gradient boosting and 0.002 under logistic regression, declines of 85% and 97%. Varying the baseline alone reproduced the size of the published range within one cohort. Note-processing tools usually need adaptation to each hospital’s documentation, so ICU and hospital leaders weighing that investment need to know what a reported gain was measured against.