Evaluating ChatGPT’s Triage and Diagnostic Capabilities in Patients Presenting with Common Causes of Foot and Ankle Pain

Research Type: Level 5 - Case report, Expert opinion, Personal observation Introduction/Purpose: ChatGPT has shown an ability to provide treatment recommendations for common orthopaedic conditions in accordance with AAOS clinical practice guidelines, including pathology specific to the foot and ankle. Previously, ChatGPT has shown great potential as a clinical support tool when triaging patients with various causes of knee pain. However, no study has looked at ChatGPT’s ability to act as a central scheduling tool that triages patients to a provider. Thus, this study explored ChatGPT’s ability to synthesize differential diagnoses and triage patients into the proper healthcare settings (Primary Care, Specialists, and Emergent Care). ChatGPT-4’s ability to identify primary diagnoses and generate treatment plans was also analyzed. Methods: 24 foot and ankle complaints warranting triage and expanded clinical scenarios were input into ChatGPT-4, with memory cleared prior to each new input to mitigate bias. In each conversation, role prompting (addressing the Artificial Intelligence (AI) interface as “Dr. AI, a fellow foot and ankle trained orthopaedic surgeon”) was employed to maximize the quality of content provided. For the 12 triage complaints, ChatGPT-4 was asked to generate a differential diagnosis and provide a decision on central scheduling (Primary Care Physician (PCP), Podiatrist/Ortho Surgeon, or Emergency Department/Urgent Care). These responses were graded for accuracy and suitability in comparison to a differential created by 2 orthopaedic foot and ankle fellowship-trained orthopedic surgeons. For the 12 clinical scenarios, ChatGPT-4 was prompted to provide treatment guidance for the patient, which was again graded. To test the higher-order capabilities of ChatGPT-4, further inquiry into these specific management recommendations was performed and graded. Results: All ChatGPT-4 diagnoses were deemed appropriate within the spectrum of potential pathologies on a differential. The top diagnosis on the differential was identical between surgeons and ChatGPT-4 for 9 (75%) of scenarios, and the top diagnosis provided by the surgeon appeared as either the first or second diagnosis in 10 (83.3%) of scenarios. Overall, 26 of 36 diagnoses (72.2%) in the differential were identical. ChatGPT’s decision for healthcare setting agreed with foot and ankle fellowship-trained orthopaedic surgeons in 6 cases (50%). When provided with expanded vignettes, the accuracy of ChatGPT-4 was maintained (75%), with the suitability of management graded as appropriate in 11 cases (91.7%). Specific information pertaining to conservative management, surgical approaches, and related treatments was appropriate and accurate in 11 cases (91.7%). Conclusion: ChatGPT-4 provided clinically reasonable differential diagnoses to triage patient complaints of foot and ankle pain, which were generally consistent with the foot and ankle physicians’ differentials. Diagnostic performance was maintained when providing additional clinical context. ChatGPT-4 shows clinically important error rates for diagnosis depending on prompting strategy and information provided; therefore, further refinements are necessary prior to full integration into clinical workflows. However, ChatGPT-4 may serve as an augment to appropriately direct patient care, potentially assisting in a system's effort to streamline scheduling.