Fluent but not fail-safe: machine-translated radiology reports still need expert review

OBJECTIVES: Cross-border care, teleradiology, and international referral pathways increasingly require radiology report translation, but isolated mistranslations may alter clinical meaning. We evaluated German-to-English machine translation of radiology reports and correction-based localization of text requiring review. MATERIALS AND METHODS: In this retrospective study, 310 de-identified German radiology reports from June and July 2025 were translated into English with DeepL. Eight radiologists assigned 5-point quality scores and corrected perceived errors. Thirty reports were reviewed by all eight readers, and 280 reports were assigned for single-reader evaluation, yielding 520 evaluations. Automated metrics and large language model judges were tested for report-level identification of translations requiring review; a correction-based segment-level edit-prediction model was evaluated on held-out shared reports. RESULTS: Most translations were rated good or excellent (339/520, 65.2%), but clinically important mistranslations involved diagnosis, anatomy, and radiologic terminology. Agreement was low for ordinal quality scores (Krippendorff α = 0.078, p = 0.006) but high for expert corrections (pairwise "bilingual evaluation understudy"-BLEU = 83.0, p < 0.001). In the conservative primary screen of 30 shared reports, no automated method discriminated translations requiring review (area under receiver operating characteristic curve [AUC] = 0.51-0.65; p = 0.100-0.920). The final edit-prediction model achieved a concordance index of 0.697 and an AUC of 0.731 for segments edited by at least six of eight readers. CONCLUSION: Machine-translated radiology reports were often acceptable on average, but translations containing errors requiring expert review persisted. Expert corrections were more reproducible than ordinal quality scores and supported localization of text requiring human review. KEY POINTS: Question Cross-border radiology workflows increasingly require report translation, but fluent machine translations may still contain meaning-changing errors requiring expert review for safe care. Findings Most machine translations were rated good or excellent, yet clinically important mistranslations occurred, and automated report-level metrics did not reliably identify reports needing review. Relevance statement Machine translation may support cross-border radiology workflows, but clinically important mistranslations in otherwise fluent reports require human-in-the-loop oversight and tools that localize text requiring expert review.