Accuracy and reliability of privacy-compliant large language models for MRI protocol selection in radiological workflows

OBJECTIVE: Accurate MRI protocolling is essential for diagnostic quality and workflow efficiency, yet manual protocol selection is time-consuming and variable. This study evaluated four contemporary open-weight large language models (LLMs) for MRI protocol and optional sequence selection from real-world clinical referrals. MATERIALS AND METHODS: In this retrospective study, 1201 consecutive referral forms (mean age 58 ± 14 years; 646 men) were provided as unaltered text to each LLM, which selected an institutional protocol and optional sequences from predefined lists. A consensus of two board-certified radiologists served as reference. Performance was assessed at protocol and sequence levels using protocol accuracy and mean Jaccard index. RESULTS: Larger general-purpose models achieved the highest concordance with expert protocolling (GPT-OSS-120B: Jaccard 0.96, 95% CI: 0.96-0.97; accuracy 0.94, 0.93-0.95; Qwen-3-235B-A22B-Instruct-2507: Jaccard 0.96, 0.95-0.97; accuracy 0.94, 0.93-0.96), outperforming MedGemma-27B-Text-IT (Jaccard 0.85, 0.84-0.87; accuracy 0.79, 0.76-0.81) and Gemma-3-27B-IT (Jaccard 0.61, 0.59-0.63; accuracy 0.58, 0.55-0.60). Unacceptable protocol selections ranged from 70/1201 cases (5.8%) for GPT-OSS and 71/1201 (5.9%) for Qwen3 to 297/1201 (24.7%) for MedGemma and 820/1,201 (68.3%) for Gemma3. Among unacceptable cases, the most common error was anatomically correct but clinically suboptimal protocol selection (57.7-70.7%), followed by incorrect anatomical region (11.6-39.4%) and optional-sequence-only errors (2.8-30.2%). GPT-OSS matched expert scan duration (p = 0.18), Qwen3 was shorter (p = 0.002), and Gemma3 and MedGemma were longer (both p < 0.001). CONCLUSION: Open-weight LLMs showed high concordance with expert MRI protocolling and may support standardized protocol selection. However, clinically relevant errors persisted, underscoring the need for radiologist oversight. KEY POINTS: Question Can open-weight LLMs support accurate and standardized MRI protocol and sequence selection from clinical referrals compared with expert reference standards? Findings In 1201 referrals, LLMs achieved high concordance with expert MRI protocol selection (accuracy up to 0.94; mean Jaccard index up to 0.96), but unacceptable errors remained model-dependent and ranged from 5.8% to 68.3%. Clinical relevance Open-weight LLMs showed high agreement with expert MRI protocol decisions from clinical text and may support standardized, reliable, and scalable MRI protocolling while maintaining radiologist oversight.