Assessing Artificial Intelligence (AI) in Patient Education: Evaluating Accuracy and Readability of Responses on Surgical Procedures for Patellar Tendon Rupture

Introduction Patients’ utilization of the internet as a resource for obtaining medical information continues to expand, with increased prevalence and access to educational materials. One method of obtaining medical information online is artificial intelligence (AI)-generated patient education materials (PEMs). As such, the medical community has a fundamental obligation to assess the accuracy, quality, and readability of AI-generated PEMs as patient resources - a critical step in promoting health literacy, combating misinformation, and, ultimately, empowering patients. Given that the perceived severity of patellar tendon ruptures (PTR) can vary, providing clear information is important to support informed decision-making. This study aimed to evaluate and compare the readability and quality of AI-generated responses to patient questions about patellar tendon repair, using four different AI chatbots: ChatGPT 3.5, ChatGPT 4, Gemini 1.0, and Perplexity. Methods There were no significant differences in readability among the four different chatbots, and they all provided responses that were better than the average American reading level. The mean DISCERN scores were as follows: Perplexity (64.2±9.2), ChatGPT 3.5 (49±7.97), Gemini 1.0 (59.2±7.43), and ChatGPT 4 (52±6.28). Even though Perplexity demonstrated the highest mean DISCERN scores among the evaluated AI models, no statistically significant differences in readability were observed among the four chatbots, although results approached significance (p = 0.075). Question 15 of the DISCERN criteria, regarding shared decision-making, was consistently rated at a high level across each AI tool, with an average rating of 4.2 out of 5. Results There were no significant differences in readability among the four different chatbots and they all provided responses that averaged above the average American reading level. The mean DISCERN scores were as follows: Perplexity (64.2±9.2), ChatGPT 3.5 (49±7.97), Gemini 1.0 (59.2±7.43), and ChatGPT 4 (52±6.28). Perplexity’s score was statistically significant when compared to ChatGPT3.5, indicating that the responses of Perplexity were more accurate and reliable than ChatGPT3.5. Question 15 of the DISCERN criteria, regarding shared decision-making, was consistently rated at a high level across each AI tool, with an average rating of 4.2 out of 5. Conclusion This study found that readability remains consistent across various AI tools, while the quality of the information may vary. Perplexity outperformed ChatGPT 3.5 in providing accurate information on patellar tendon ruptures. AI tools demonstrated variability in informational quality scores, although these differences were not statistically significant, highlighting the importance of carefully evaluating AI-generated content before using it as a patient education resource.