Use of imaging prior to orthopedic oncology referral: An analysis of ChatGPT recommendations

A standardized set of clinical practice guidelines from the Musculoskeletal Tumor Society (MSTS) was released in 2018, seeking to aid non-specialist physicians in diagnosing and generating treatment plans for bone and soft-tissue lesions. 1 Artificial intelligence (AI) models like ChatGPT, an OpenAI language modeling tool, assist physicians in treatment planning and reducing administrative burden. 2 Our objectives were to determine how closely ChatGPT recommendations align with MSTS guidelines for imaging bone and soft tissue lesions, prior to orthopedic oncology referral. We also investigated if ChatGPT clearly and accurately explained these MSTS guidelines to the user. We developed questions to assess ChatGPT’s alignment with MSTS guidelines. Answers from ChatGPT were double-blinded and evaluated for concordance. Answers were scored using four categories: accuracy, overconclusiveness, addition of supplementary information, and incompleteness. Chi-square test was used for analysis with a statistically significant p-value of less than 0.05. A total of 14 questions were generated from the 12 MSTS guidelines. Results showed alignment between the accuracy of ChatGPT’s responses and the 12 guidelines. Of the 14 questions posed to ChatGPT, 10 were deemed accurate to the guidelines. However, responses to 13 questions were deemed to be overconclusive or to contain supplementary information (p<0.05). Additionally, 9 responses were deemed incomplete when compared to MSTS guidelines (p<0.05). The concordance between the accuracy of responses and the 2018 MSTS guidelines is convincing, suggesting that imaging methods recommended by MSTS to front-line practitioners are generally aligned with AI-based modeling recommendations. However, it is not yet sufficient as a standalone modality, often providing supplementary or incomplete information. While AI-based predictions in healthcare are an evolving modality, their value as a diagnostic tool demands further rigorous research.