RadCoT: a Radiological Chain-of-Thought framework for enhanced error detection in radiology reports

Abstract Background Errors in radiology reports are a major patient-safety concern and are difficult to detect with manual quality assurance (QA). Large language models (LLMs) can assist, but generic prompting does not reflect radiologists’ structured, section-based workflows. Objective To develop and evaluate RadCoT (Radiological Chain-of-Thought), a domain-specific prompting framework aligning LLM reasoning with radiological review workflows, and to assess whether it enables open-source models to approach commercial benchmarks for error detection. Materials and methods In this retrospective study, 1,170 clinician-validated errors were extracted from a departmental QA repository (January 2021–December 2024), corresponding to 900 error-containing reports. An additional 300 error-free reports served as controls, yielding 1,200 reports balanced across radiography, ultrasound, CT, and MRI. Errors were categorized into five types by experienced radiologists. Seven LLMs were evaluated using standard prompting and the six-step RadCoT framework. Micro-averaged precision, recall, and F1 were computed at the error-instance level. Error type, modality-specific performance, and inference time were analyzed. Results RadCoT significantly improved the mean micro-averaged F1 across all models from 0.77 ± 0.06 (standard) to 0.85 ± 0.06 (RadCoT; p = 0.003). GPT-4o with RadCoT achieved the highest F1 (0.93). Llama-3.3-70B with RadCoT (F1 = 0.89) narrowed the gap with GPT-4o under standard prompting (F1 = 0.88; paired t -test, Holm-adjusted p = 0.28). Interpretation errors showed the largest gain, with F1 improving from 0.57 to 0.75. Conclusion RadCoT consistently enhances LLM-based error detection, particularly for complex logic and consistency errors, closing the gap between open-source and commercial models and offering a pathway for privacy-preserving, on-premises QA. Key Points Question Manual review misses radiology report errors and remains difficult to scale. Findings Structured prompting improves detection of interpretation and section-consistency errors across seven models. Relevance statement Open-source models approach commercial performance for privacy-preserving on-premises radiology quality assurance.