Comparing Large Language Models and Traditional Machine Translation Tools for Translating Medical Consultation Summaries – A Pilot Feasibility Study (Preprint)

<sec> <title>BACKGROUND</title> Translation of medical consultation summaries is essential for equitable healthcare communication in culturally and linguistically diverse (CALD) populations. While machine translation (MT) tools and large language models (LLMs) are widely accessible, their feasibility and safety for healthcare contexts remain underexplored. </sec> <sec> <title>OBJECTIVE</title> This pilot study investigates the feasibility of using LLMs and traditional MT tools to translate medical consultation summaries from English into the most common languages other than English spoken in Australia – Arabic, Chinese (simplified written form), and Vietnamese - the three most common languages other than English in Australia. </sec> <sec> <title>METHODS</title> Two simulated summaries -a simple, patient-facing summary and a complex, clinician-orientated interprofessional letter - were translated using three LLMs (GPT-4o, LLAMA-3.1, GEMMA-2) and three MT tools (Google Translate, Microsoft Bing Translator, DeepL). Translations were benchmarked against professional third party interpreter translations using BLEU, CHR-F, and METEOR metrics. </sec> <sec> <title>RESULTS</title> Translation performance varied across languages, tools, and summary complexity. Traditional MT tools outperformed LLMs on surface-level metrics, while LLMs showed relative strengths in semantic similarity for Vietnamese and Chinese. Arabic translations improved with complex input, suggesting morphological advantages. Metric-based evaluation highlighted feasibility but also risks, particularly in Chinese clinical contexts. </sec> <sec> <title>CONCLUSIONS</title> This pilot provides formative evidence of opportunities and limitations in applying AI translation for healthcare communication. Findings underscore the importance of human oversight, domain-specific evaluation metrics, and further formative and clinical research to guide safe, equitable use of AI translation tools. </sec>