Keeping private patient data off the cloud: A comparison of local LLMs for anonymizing radiology reports

To evaluate the performance of open-source, locally deployed large language models (LLMs) in anonymizing radiology reports while preserving clinical data, ensuring compliance with privacy regulations like the Health Insurance Portability and Accountability Act (HIPAA), without relying on cloud-based solutions. Seven state-of-the-art open-source LLMs—Qwen-2.5-coder (7B, 32B), Llama v3.1 (8B, 70B), Llama v3.3 (70B), and Phi3/4 (14B)—were tested on 1000 randomly selected radiology reports (CT, MRI, PET/CT) from a cancer imaging cohort. Models were tasked with generating regular expression (RegEx) rules to remove protected health information (PHI) as defined by HIPAA. Inference was conducted on an A100 GPU and an M4 Max MacBook Pro using the ollama framework. Performance was assessed by three readers for protected health information (PHI) removal accuracy and preservation of non-PHI clinical data, with runtimes recorded. Qwen models outperformed others, with Qwen7B achieving a 97.59 % success rate in preserving clinical data (vs. 24.33 % for Llama8B) and Qwen32B achieving 100 % PHI removal (vs. 1.91 % patient names missed by Llama v3.1 70B). Llama models frequently misidentified clinical numbers as PHI, while Phi models produced unusable results due to overly aggressive rules or hallucinations. Runtimes were significantly shorter for Qwen models (e.g., Qwen7B: 2h50′ on A100 vs. Llama8B: 3h30′). Locally deployed Qwen-2.5-coder models offer a superior, privacy-preserving solution for anonymizing radiology reports, balancing PHI removal with clinical data retention. This approach supports scalable, compliant anonymization of radiology reports for research without cloud dependency, outperforming other current state-of-the-art local LLMs. • Locally deployed open-source LLMs, such as Qwen 2.5-coder, enable HIPAA-compliant anonymization of radiology reports while maintaining data privacy. • Qwen models outperform Llama and Phi models in achieving complete PHI removal with minimal impact on non-PHI clinical data. • This study demonstrates the feasibility and efficiency of using locally run LLMs, fostering broader accessibility and accelerated medical research.