Assessing Healthcare Stakeholder Understanding of Machine Learning Documentation

Artificial Intelligence (AI) has significantly advanced clinical decision support systems in healthcare, particularly using Machine Learning (ML) models. However, the technical nature of current ML model documentation often leads to lack of comprehension among healthcare stakeholders. This study assessed the understanding of ML model documentation for a sepsis prediction model during an interdisciplinary workshop with 24 participants. Results showed that participants had moderate AI literacy (mean score: 40.13 out of 65) and that on average 65% of the model documentation was understood. Key barriers to understanding included technical jargon, unexplained abbreviations, lack of contextual clarity, and dense data presentation, which impacted usability. Participants recommended the use of simplified language, clear explanations, and visual aids to enhance clarity and facilitate integration into clinical workflows.