Transcription accuracy of automatic speech recognition for clinical orthodontic

Accurate clinical records are fundamental to dental practice. Automatic Speech Recognition (ASR) has the capacity to convert spoken clinical language into written text within the electronic health record; however, accuracy of ASR in natural language processing for clinical dentistry remains uncertain. The aim of this study was to investigate transcriptional accuracy of ASR systems using orthodontic clinical records as the experimental model. Specifically, we utilised 4 commercial ASR systems (Heidi-Health, DigitalTCO, DragonMedicalOne, DragonAnywhere); 5 Application Programming Interfaces (Amazon, Google, Speechmatics, Whisper, GPT4oTranscribe); and a 2-stage pipeline coupling GPT4oTranscribe with the GPT4o Large Language Model (LLM) for generative error correction (GPT4oTranscribeCorrected). Orthodontic diagnostic and treatment-planning summaries (n=200; 10 subject-domains; 43,408 words; 6 hours-audio) were narrated and recorded for analysis. Primary outcome was Domain Word Error Rate (DWER) which investigates clinical-terminology transcription errors against the SNOMED-CT database. Secondary outcomes included WER, Non-Domain (N-D)WER, lexical accuracy (ROUGE-score), semantic similarity (BERT; BART-score), hallucinations (transcribed text not in the spoken input) and qualitative error-analysis. GPT4oTranscribeCorrected was transcriptionally most accurate (DWER=3.5%; WER=3.7%) with DWER decreasing by 54.9% versus GPT4oTranscribe. Heidi-Health was the highest-performing commercial-system (DWER=6.2%; WER=5.4%) with DragonAnywhere worst (WER=33.9%). All systems were less accurate with technical vocabulary (DWER>N-DWER;P<0.001) except GPT4oTranscribeCorrected. Significant differences were seen across systems for ROUGE, BERT and BART-scores (P<0.001). Based on post-hoc pairwise comparisons GPT4oTranscribeCorrected performed best and DragonAnywhere was consistently worst for lexical and semantic-errors. Hallucinations were absent except for Whisper (n=57) and DigitalTCO (n=1). Across systems, background-noise increased DWER and WER (P<0.001). Importantly, clinically-significant errors were seen with all systems, ranging from 2-66% (GPT4oTranscribeCorrected-clean; DragonMedicalOne-background-noise, respectively). Variation in narrator accent had no effect in clean conditions (P=0.65) and small effect with background-noise (P=0.001). ASR systems deliver single digit transcription error-rates, particularly when coupled with LLM-based correction, but clinically-significant errors persist. Verification of clinical records is essential when using current ASR systems.