Impact of acoustic and informational noise on AI-generated clinical summaries

Objectives To investigate the effect of introducing environmental noise, microphone identity and position, and informational noise on the accuracy of a commercial Clinical AI Scribe (CAIS). Methods and analysis Consultations on five medical conditions (memory loss, diarrhoea, headaches, skin rash or prostate symptoms) were recorded in a simulated primary care setting. A ground-truth audio file was produced, along with a ground-truth clinical summary via the CAIS. Different types of ‘noise’ at different levels were then investigated by (i) varying the type and distance of the microphone, (ii) introducing background noise (rain, baby, construction, toddler) at various levels and (iii) introducing informational noise (irrelevant discussions, medical and non-medical). These were then compared with the ground-truth summaries. The degradation of clinical summary accuracy was measured (omissions, hallucinations and inclusions), relative to the ground-truth summary. Results Errors increased for all noise levels and noise types, principally being omissions. At 4.5 m from the consultation, for all microphones tested, all facts were omitted. At up to 2 m, all microphones, except the laptop, only introduced a small number (<5) of omissions. For background noise, the type of noise was found to correlate with omissions, with toddler and heavy rain being particularly deleterious. The CAIS was remarkably apt at rejecting additional informational noise. Conclusions The CAIS generally excelled at capturing a doctor-patient consultation and producing accurate clinical summaries; however, error rate (especially omissions) increased notably when acoustic noise was introduced. The system was better able to handle informational noise.