Safety and efficacy of privacy-preserving models to create Lay summaries of brain MRI reports

Patient access to radiology reports has heightened the need for patient-friendly communication. Automated generation of patient-centered summaries using large language models (LLMs) is a promising solution. However, their use on real-life reports is limited by privacy concerns. Here, we evaluate the safety and effectiveness of on-premise, privacy-preserving LLMs for generating lay summaries of real French brain MRI reports for emergency presentations of headache. In this retrospective study, we sampled 105 brain MRI reports (January–December 2022) for radiologist evaluation and a subset of 30 reports for non-physician evaluation. Three open-weights models (Llama 3.3 70B, Athene V2, Mistral Small) generated French lay summaries via a single standardized prompt. Radiologists’ mean ratings across models were high for exactness (4.10, 95% CI: 4.04–4.16), exhaustiveness (4.34, 95% CI: 4.29–4.39), didacticness (3.83, 95% CI: 3.79–3.88), and readiness for clinical use (3.84, 95% CI: 3.79–3.89). Non-physicians reported higher perceived understanding with summaries, from 2.85 (95% CI: 2.67–3.04) to 4.27 (95% CI: 4.15–4.38, p < 0.001). The correct identification rate for reports increased from 75.2% to 83.6% (p < 0.001). The ability to identify causal findings also improved, from 80.6% to 84.8% (p < 0.001) overall. Overall error rate in LLM-generated lay summaries was 19.7% (62/315), warranting expert oversight.