Artificial Intelligence Chatbots Can Build An Accurate Differential For Acute Musculoskeletal Injuries

As the utilization of Artificial Intelligence (AI) becomes more prevalent, there are widespread implications for its use in healthcare. Research indicates early adoption of AI chatbots by patients in self diagnosis; its expanding use necessitates evaluation of the accuracy of its responses. PURPOSE: To determine the accuracy and limitations of chatbots in building a differential for common acute musculoskeletal injuries in comparison to an established medical library, MedlinePlus. METHODS: Four questions addressing what causes common musculoskeletal pain (neck, shoulder, knee, low back) were created. Three chatbot responses to these questions were gathered (ChatGPT 3.5, Google Bard, Bing Chat Creative) to compare to MedlinePlus. Medline plus was considered to be trusted because of its information being sourced from the National Institute of Health. Any associated hyperlinks were broken to avoid the participants from knowing which source they were grading. The responses were blinded, randomized, and subsequently graded by five sports medicine trained physicians regarding accuracy on a 1-4 scale. The data was compared using Welch’s two-sample t-test with Satterthwaite approximation for degrees of freedom. RESULTS: ChatGPT 3.5 ranked the highest in response accuracy (x̄=3.175, s = 0.374) followed by GoogleBard (x̄=2.875, s = 0.609), Bingchat (x̄=2.8, s = 0.682), and MedlinePlus (x̄=2.55, s = 0.558). MedlinePlus only scored better than one chatbot in a single category. ChatGPT 3.5, while maintaining the largest differential to MedlinePlus, did not significantly score better than its counterpart (t(8) = -2.082, p = 0.071). CONCLUSION: Although this study identified a trend in which AI chatbots, including ChatGPT 3.5, Google Bard, and Bing Chat Creative, achieved higher accuracy scores than MedlinePlus in building differentials for musculoskeletal injuries, this difference was not statistically significant. Notably, MedlinePlus, a widely accepted and reliable patient-facing medical library, showed comparable effectiveness to these chatbots. These findings highlight the rapidly advancing capacity of chatbots in healthcare; AI chatbots may be useful in providing insight to patients about common causes of musculoskeletal pain. There was no grant funding. AI Chatbots as described in the methods were used to generate responses to common complaints. These results were blinded and compared the MedlinePlus by physicians. No chatbots were used in creating the content of the abstract itself.