Physician Gestalt Compared With Artificial Intelligence Model to Predict Intubation in Critically Ill Patients

IMPORTANCE: Accurate prediction of intubation in critically ill patients could enable interventions that improve patient outcomes. However, the performance of intensive care physicians compared with machine learning (ML) models remains unknown. OBJECTIVES: To investigate intensive care physicians' ability to predict the need for intubation within 24 hours and compare their performance against an established ML model, Vent.io. DESIGN, SETTING, AND PARTICIPANTS: This prospective observational study in two ICUs surveyed intensivists to test their ability to predict the need for intubation of adult patients under their care. Physician predictions of intubation were then compared with predictions from Vent.io. MAIN OUTCOMES AND MEASURES: Primary metrics include prediction sensitivity, specificity, and descriptive statistics for both physicians and ML model. Generalized linear mixed models investigated the fixed effect of the predictor (physician vs. Vent.io) on both sensitivity and specificity while accounting for the random effects from different physicians. Similar modeling was used to investigate the relationship between physician confidence and correctness. RESULTS: Overall, physicians were quite confident in their predictions of intubation with a median score of 8 (on a 0-10 point scale, with 0 being not at all confident and 10 being extremely confident) out of the 302 surveys administered. Sensitivity was 0.190 and 0.714 for physicians and Vent.io, respectively. Specificity was 0.960 and 0.673 for physicians and Vent.io, respectively. Generalized linear mixed modeling showed that physician confidence was associated with greater odds of correctly predicting intubation outcome (odds ratio [OR] 1.49; 95% CI, 1.22-1.84; p < 0.001). Vent.io had significantly greater odds of being correct when patients required intubation compared with physicians (OR 18.68; 95% CI, 1.87-186.31; p = 0.013). However, intensive care physicians outperformed Vent.io at correctly predicting when patients did not require intubation (OR 24.80; 95% CI, 13.22-46.52; p < 0.001). CONCLUSIONS AND RELEVANCE: Although predictive performance compared with human experts is promising, Vent.io needs real-time testing in a randomized clinical trial to determine if its deployment can improve clinical outcomes.