Benchmarking Open-Source Large Language Models for Breast Ultrasound Report Processing: A Multicenter Study

Background/Objectives: General-purpose large language models (LLMs) may support medical reporting, but their role in breast ultrasound (BUS) workflows remains uncertain. We evaluated locally deployable open-source LLMs for BUS report quality control, BI-RADS reasoning, and clinical decision support tasks under physician supervision using retrospective, multicenter reports from real-world clinical practice. We further characterized their relative strengths, limitations, and potential assistive roles compared with proprietary models and ultrasound physicians. Methods: We built a multicenter benchmark of 1173 task-specific Chinese BUS reports from three tertiary hospitals. Seven tasks covered report quality control, BI-RADS reasoning and clinical decision support. We evaluated 24 open-source LLMs, 20 proprietary LLMs and six ultrasound physicians. Tasks 1–5 used reference-standard classification metrics. Tasks 6–7 used blinded expert ratings on five-point Likert scales. Results: Open-source LLMs did not differ significantly from proprietary LLMs in basic error detection, findings–impression inconsistency detection, treatment response assessment or follow-up recommendation generation. They performed worse in terminology non-standardization detection, single-modality BI-RADS categorization and multimodal BI-RADS categorization. Mean accuracies were 0.3775 vs. 0.4563, 0.4121 vs. 0.4920 and 0.6149 vs. 0.7178, respectively. In the 210-case human-evaluation subset, physicians outperformed LLMs in basic error detection, findings–impression inconsistency detection, single-modality BI-RADS categorization and multimodal BI-RADS categorization. LLMs received higher expert-rated scores for treatment response assessment and follow-up recommendation generation. Terminology non-standardization detection showed mixed results across physician seniority and model group. Among open-source LLMs, reasoning-capable models outperformed non-reasoning models in six of seven tasks. The latest open-source release period showed higher composite performance, although substantial model-level heterogeneity remained. Conclusions: Open-source LLMs may support selected BUS report-processing tasks, especially longitudinal summarization and recommendation drafting. Their weaker performance in terminology non-standardization detection and BI-RADS reasoning indicates persistent limits in specialized diagnostic reasoning. Clinical deployment should remain physician-supervised and should include local validation, version control, and task-specific model selection.