Safety and quality of patient-facing frozen shoulder information from four generative AI chatbot interfaces: a cross-sectional study

Abstract Patients increasingly use generative artificial intelligence (AI) chatbots for musculoskeletal health information, but their safety and suitability for frozen shoulder education are uncertain. We submitted 50 prespecified patient- and caregiver-facing questions once to each of ChatGPT, Gemini, DeepSeek, and Doubao on 17 July 2026, yielding 200 first responses. Five clinicians independently assessed deidentified responses against question-specific evidence standards for safety, accuracy, empathy, information quality, transparency, and global quality; readability was calculated computationally. Twenty-one responses (10.5%) were classified as potentially unsafe. Interface-specific proportions ranged from 6.0% to 14.0%, without a significant overall difference (P = 0.442). Accuracy differed across interfaces (P < 0.001; Kendall's W = 0.639), as did empathy (P < 0.001; W = 0.580). ChatGPT had the highest median accuracy (4.40), DISCERN (60.00), and JAMA benchmark (3.00) scores, whereas Gemini had the highest median binary-adapted EQIP score (83.00) and the most favorable readability profile. All interfaces exceeded commonly recommended reading levels. No interface was uniformly superior: each produced potentially unsafe content and limited response-level transparency. These tools may support patient education but should not replace individualized clinical assessment and guidance.