Diagnostic accuracy & clinical importance of AI confidence for extremity fracture detection: 2,508-patient retrospective cohort

Abstract Purpose To estimate diagnostic performance of a deep learning algorithm for extremity fracture detection on radiographs in patients aged ≥ 2 years using a refined reference standard. Secondary, to compare positive predictive value (PPV) by the algorithm’s built-in confidence level (high vs. low) and diagnostic performance between adults (≥ 18 years) and children. Methods This retrospective single-center study consecutively included patients with radiography of a suspected extremity fracture between January and December 2024. The index test was the algorithm output (bounding boxes with confidence). The reference standard was the radiology report with discordant cases adjudicated by follow-up imaging when available or specialist review. Sensitivity, specificity, PPV and negative predictive value (NPV) were calculated with 95% confidence intervals; differences were tested using the McNemar and a score test and patient-clustered logistic regression. Results There were 2,508 patients, median age 34 years (range 2–105), 1,236 males; 815 patients had a total of 1,028 fractures. Per-fracture sensitivity and PPV were 92.7% (95% CI: 90.9–94.5) and 87.6% (95% CI: 85.5–89.7); Specificity and NPV were 94.4% (95% CI: 93.2-95.4) and 96.6% (95% CI: 95.6–97.4). PPV for high-confidence was superior to low-confidence detections (99.1% vs 59.1%, p < 0.001). Children had higher per-fracture PPV than adults (91.7% vs. 86.1%; p = 0.01), with no significant difference in per-fracture sensitivity, or case-wise performance. Conclusion The algorithm showed good diagnostic performance for extremity fracture detection on radiographs. The AI’s confidence stratification strongly influenced PPV, and higher PPV was seen among children.