Development and usability testing of a machine learning clinical decision support tool for suggesting etiologies and treatment of clinical deterioration

BACKGROUND: Early warning score clinical decision support (CDS) systems identify patients at risk of clinical deterioration, but do not provide insights on the underlying cause. Machine learning (ML) models may address this limitation, but how to convey such outputs to clinicians remains unclear. This study elicited clinicians' design requirements and assessed their impact on ML-CDS usability in a controlled lab setting. METHODS: This sequential exploratory mixed-methods approach began with focus groups to obtain design requirements for the ML-CDS interface. After revising the prototype, we conducted usability tests with critical care or hospital medicine clinicians to assess effectiveness, efficiency, and satisfaction. Usability tests involved an experimental EHR interface with the ML-CDS tool and a control EHR interface without it. Participants reviewed five scripted patient cases per interface with a washout period. Wilcoxon signed-rank tests were used to compare usability outcomes between the interfaces. RESULTS: Focus groups included 26 clinicians. Design requirements included understanding the rationale for ML-CDS recommendations, assessing how EHR data quality issues affect recommendation accuracy, consolidating key information on a single screen, graphically visualizing information, and integrating ML-CDS tools into pre-existing care pathways. The usability tests (n = 23) revealed that the experimental interface was associated with improvements in diagnostic accuracy, perceived workload, and satisfaction compared to the control interface (all p < 0.05) in a controlled lab setting. CONCLUSIONS: This study identifies design requirements to improve ML-CDS acceptability. Our controlled usability tests suggest that incorporating these requirements may improve overall usability.