Smart Answers, Hard to Read: Accuracy and Readability of Chatbots on FAQs Regarding Osseointegration for Limb Prosthetics - A Mixed Method Study

Introduction: Large language models have transformed how individuals access and utilize information, demonstrating remarkable potential to influence patient care and clinical decisions. Osseointegration (OI) for limb reconstruction has garnered extensive attention due to its advantages over traditional socket-type prostheses. However, as a novel technique, high-quality publicly accessible patient information is limited. This study aimed to assess and compare the quality of responses from two common chatbots regarding frequently asked questions on Osseo-integrated prostheses for limb reconstruction. Materials and Methods: Ten comprehensive questions on OI spanning five categories (i.e., basic information, contraindications, OI procedure and implant, perioperative care and physical therapy, and complications and outcomes) were developed. These questions were entered into ChatGPT and Gemini in separate sessions, and responses were recorded. Twelve experts, including orthopedic surgeons, prosthetist, physiatrists, and physical therapists, rated the responses. Readability was evaluated using the Flesch-Kincaid Reading Ease Score and Grade Level scoring system. Results: Both chatbots provided satisfactory responses requiring minimal clarification (ChatGPT, 2.1 ± 0.37; Gemini, 2.3 ± 0.24; P = 0.05). Experts preferred ChatGPT responses for 70% of the questions over Gemini responses. There was no difference in ratings based on expert occupation for both ChatGPT and Gemini responses ( P = 0.16 vs. 0.57). Both chatbots produced responses with “very difficult” readability (ChatGPT, 20.54 ± 8.40; Gemini, 20.53 ± 7.24) and a college graduate reading level (ChatGPT, 15.42 ± 1.72; Gemini, 14.75 ± 1.35). Conclusions: Online chatbots can provide accurate responses to questions regarding OI for limb reconstruction, but they have limited readability. However, the complexity of their language underscores the need for healthcare providers to interpret and communicate this information to patients clearly. Clinical Relevance: ChatGPT and Gemini can provide satisfactory information regarding the topic of osseointegration; however, the high reading difficulty emphasizes the importance of clinicians in translating and conveying this information effectively to patients.