Benchmarking Off-the-Shelf ASR Systems on Polish Radiology Dictation: Terminology, Numbers, and Clinically Critical Errors
Ready-to-use automatic speech recognition (ASR) systems are increasingly considered for dictating radiology reports, yet their behaviour on Polish clinical speech is poorly characterised. We benchmark 19 off-the-shelf systems (12 open configurations and 7 commercial cloud services) on 939 natural-voice recordings (2.69 h, 14 speakers) of thyroid ultrasound and head-and-neck radiology. Beyond the word error rate (WER) and character error rate (CER), we measure the correctness of radiology terminology (term error rate, TER) against general words (GWER), and errors in numbers, negations, and body side. In classical ASR, radiology terms are recognised 3–6 times worse than general words; LLM-based transcribers narrow this gap. The best open model (Whisper large-v3) misrecognises 15.8% of term occurrences but 3.1% of general words; the best cloud system (OpenAI gpt-4o-transcribe) reaches WER 5.96%, and Google Gemini 3.5 Flash has the lowest TER (6.6%). Number-formatting conventions can inflate raw WER substantially, which is why a number-normalised WER is reported alongside.
Authors
- Konrad Strojny (ORCID: https://orcid.org/0009-0006-9255-0975)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.5281/zenodo.24339276
- Primary Topic
- Speech Recognition and Synthesis
- Type
- article
- Field-Weighted Citation Impact
- 0.00