An evaluation of ambient artificial intelligence scribes at two academic medical centers

Ambient artificial intelligence (AI) documentation technologies aim to address challenges with clinical documentation. While earlier studies suggested that these tools may improve documentation workflows and reduce provider burden, most studies have been single-center and small in size. We present a comprehensive, multi-site assessment of Dragon Copilot at two academic medical centers. The objectives of this study were to evaluate clinical usability, documentation quality, and clinician satisfaction associated with the use of an AI Scribe. We deployed a cross-sectional survey study with two surveys. Participants were ambulatory faculty physicians or advanced practice providers (166 total). Clinicians represented four different academic ranks and were stratified by specialty into three groups. The surveys consisted of custom questions and validated instruments (Usability Metric for User Experience [UMUX-LITE], System Usability Scale [SUS], Physician Documentation Quality Instrument [PDQI], Net Promoter Score®[NPS®]). We present descriptive statistics for the responses from the two institutions, with differences quantified using the standard mean difference (SMD). We analyzed 120 responses. The mean SUS score was 82 (Range = 0 to 100; SD = 16; SMD = 0.19). Documentation quality evaluated by PDQI found a mean of 36.91/50 (SD = 7.78; SMD = 0.29). Clinician satisfaction with the AI scribe, as measured by NPS, was 30 overall (Range = − 100 to 100). Overall response across both medical centers was mixed. Clinicians felt the AI scribe fit well into their clinical workflow, reporting an improved patient encounter, less documentation time, and improved focus on patients’ needs. Challenges faced by clinicians (burnout, EHR frustration, work/life balance, cognitive burden) were not eliminated by the AI scribe, but it had a substantial effect on clinician satisfaction and well-being. In this first known multi-site evaluation of an ambient AI scribe, Dragon Copilot demonstrated high usability and some favorable clinician-reported outcomes. These findings provide important guidance for health systems considering ambient documentation technologies.