AI in Diagnostic Radiology: What Happens When Algorithms Are Updated

Background: Interpretation of radiographs is prone to diagnostic errors. Artificial intelligence (AI) has shown promising results in fracture detection, although systematic evaluation of software updates remains limited. This study compares the diagnostic performance of two versions of an AI-based fracture detection software in hand and ankle radiographs and assesses the influence of AI output on diagnostic decisions. Methods: This retrospective diagnostic accuracy study included 193 hand and ankle examinations obtained during routine clinical practice at Lillebaelt Hospital, Denmark. Radiographs were analysed using two versions of the same AI software and compared with the diagnostic report as the reference standard. Diagnostic performance of both versions was assessed using diagnostic accuracy metrics. Exploratory subgroup analyses were conducted to further investigate the difference in performance. The influence of AI was evaluated by the proportion of reports revised after review of AI output. Results: The newest software version demonstrated higher diagnostic performance than the older one (accuracy 0.933 vs. 0.824; p < 0.001). Similar improvements were observed across patient subgroups. Excluding radiographs containing casts resulted in only minimal changes in performance (accuracy in version 2: 0.930 vs. 0.933). In 8 of 15 discordant cases, reporting radiographers revised the initial assessment upon reassessment. Conclusions: The newest version demonstrated higher overall diagnostic performance, indicating that software updates can enhance the accuracy of AI-assisted fracture detection. The proportion of revised assessments suggests that radiographers’ decisions may be influenced by AI output.