Concordance of ChatGPT with a Fellowship-Trained Orthopaedic Surgeon in Detecting Periprosthetic Osteolysis Following Total Shoulder Arthroplasty

Purpose This study aimed to evaluate the agreement between ChatGPT 4.0 and a fellowship-trained orthopaedic surgeon in detecting periprosthetic osteolysis on radiographs following anatomic total shoulder arthroplasty (aTSA), using the Gruen and Modified Gruen zone scoring systems. Agreement with a fellowship-trained orthopaedic surgeon was evaluated. Methods A total of 73 radiographs from patients with failed aTSAs were retrospectively analyzed. Each radiograph was assessed for osteolysis across predefined zones using both traditional and modified Gruen zone scoring. ChatGPT-4.0 received zone definitions and reference images before evaluating each zone as positive (1) or negative (0) for osteolysis. A stem was labeled “loose” if ≥2 zones showed osteolysis. A fellowship-trained orthopaedic surgeon independently scored the same radiographs. Agreement was evaluated using Cohen’s kappa, and agreement metrics relative to the surgeon were calculated. Results Agreement metrics relative to the surgeon demonstrated variable sensitivity across zones, ranging from 33% to 100%, with consistently high specificity (55% to 100%). The highest agreement was observed in Modified Gruen M2 (κ = 1.000), followed by Modified Gruen Loose (κ = 0.812), Gruen 7 and Gruen 8 (κ = 0.754), Gruen 1 (κ = 0.734), Gruen 6 (κ = 0.686), Overall Loose (κ = 0.718), and Gruen Loose (κ = 0.670). Moderate agreement was observed in Gruen 3 (κ = 0.456) and Modified Gruen L2 (κ = 0.437). Fair agreement was observed in Modified Gruen US (κ = 0.395) and Gruen 5 (κ = 0.333), while slight to poor agreement was observed in Gruen 4 (κ = 0.183), Modified Gruen M1 (κ = 0.161), and Modified Gruen L1 (κ = −0.067). Overall, substantial or greater agreement was observed in 5 of 8 Gruen zones (62.5%) and 1 of 5 Modified Gruen zones (20%). Agreement relative to the surgeon ranged from 50% to 100%, with most zones exceeding 85%. Conclusion ChatGPT 4.0 demonstrates variable agreement with a fellowship-trained orthopaedic surgeon in detecting periprosthetic osteolysis, particularly in zones with well-defined criteria. However, agreement remained inconsistent across anatomical zones. Since the cohort consisted exclusively of failed anatomic total shoulder arthroplasties from an implant retrieval database, these findings may overrepresent the prevalence of osteolysis and limit generalizability. Level of Evidence Level III, Retrospective Comparative Study.