Institution-specific CT protocol assignment from referral text: A comparison of prompt-engineered and fine-tuned transformer-based language models

Purpose Radiological vetting involves reviewing imaging referrals to confirm their appropriateness and assign an acquisition protocol. Growing workforce shortages have turned this vital step into a resource-intensive bottleneck. While Large Language Models (LLMs) have shown promise in applying generic appropriateness criteria, their ability to adapt to institution-specific protocols remains unexplored. This study evaluates a generative LLM (GPT-4o-mini) and a medically pretrained RoBERTa model on institution-specific CT protocol assignment from referral text based on local clinical guidelines Methods In this retrospective study, CT requests at our institution from 2023 were categorized into 17 protocol labels, focusing on thoracic and urogenital examinations. We evaluated GPT-4o-mini and a fine-tuned RoBERTa model against the original vetting by radiologists. Primary outcomes included overall agreement and per-class performance metrics. We compared the model performance using McNemar's test and estimated the effect sizes using odds ratios. Results Among 16,926 referrals, RoBERTa achieved a higher overall agreement with the reference standard (95.7%, 736 mismatches) compared to GPT-4o-mini (95.2%, 802 mismatches). Both models performed well for frequent protocols, with renal stone and low-dose chest CTs achieving PPV ≥0.98. Macro-averaged metrics showed a larger gap between the models: balanced accuracy was 86.1% for RoBERTa versus 70.1% for GPT-4o-mini, and macro F1 was 74.7% versus 59.8%, respectively. Conclusion Both fine-tuned and prompt-based models demonstrated high adherence to the local vetting guidelines. The fine-tuned RoBERTa model performed better on most low-frequency protocols (macro F1 74.7% vs 59.8%), whereas the prompt-engineered approach offers a flexible alternative that avoids per-institution retraining.