Assessing ChatGPT Ability to Answer Common Patient Questions on Distal Biceps Ruptures

Background ChatGPT (OpenAI) is an artificial intelligence (AI) driven software that responds conversationally to written prompts. Given the rapid growth of ChatGPT and its accessibility, it is likely that patients may use this technology for medical education. The aim of this study was to determine if ChatGPT could appropriately respond to frequently asked questions (FAQ) pertaining to distal biceps rupture. Methods Ten questions were gathered from the ‘Frequently Asked Questions’ section of ten well-known health care institutions. The questions were then entered into ChatGPT 3.5 with no follow-up prompts. The responses were analyzed for accuracy, completeness, and readability using DISCERN, Journal of the American Medical Association (JAMA) Benchmark Criteria, and Flesch-Kincaid Grade Level scores. Results The average Flesch-Kincaid Grade Level of the ten responses was 14.9 (range: 11.87-17.44). Using the DISCERN score, one response was rated as “Very Poor,” seven responses were rated as “Poor,” and two responses were rated as “Fair.” The JAMA Benchmark criteria was zero for all responses. Conclusion The chatbot was able to answer commonly asked questions by patients pertaining to distal biceps rupture sufficiently, responses in a coherent manner that most patients would be able to understand. However, the high reading level may not be suitable for most patients, and the lack of documentation of the sources of information prevents readers from checking the factual content of the responses. For a comprehensive discussion of the treatment options and their implications, it remains essential for patients to have a consultation with a board-certified orthopedic surgeon with the appropriate training to address this injury.