Safety, accuracy, empathy, reliability, and readability of large language model responses to patient health questions about adenomyosis: A comparative study

BACKGROUND: Patients increasingly use large language model (LLM)-based chatbots to seek health information. Adenomyosis is a common gynecological condition associated with heavy menstrual bleeding, dysmenorrhea, pelvic pain, infertility, and pregnancy-related concerns. The quality and safety of chatbot-generated patient information is clinically relevant. OBJECTIVE: Our objective was to compare the safety, accuracy, empathy, reliability, information quality, and readability of LLM responses to patient health questions about adenomyosis. METHODS: This comparative cross-sectional evaluation was designed and reported according to the CHART checklist for chatbot health advice studies. Standardized chatbot queries were posed from June 13 to June 15, 2026, in Wuhan, China. A 40-item adenomyosis patient question set was submitted to four LLM-based chatbots: ChatGPT, Gemini, DeepSeek, and Doubao. Five trained gynecology experts evaluated the safety, accuracy, empathy, DISCERN score, EQIP (Ensuring Quality Information for Patients) score, Journal of the American Medical Association (JAMA) benchmark criteria, and Global Quality Scale score using prespecified criteria; readability was calculated algorithmically using automated text-processing procedures. Safety was coded as 0 for safe and 1 for potentially unsafe. Inter-rater agreement was assessed using Fleiss' kappa for safety and intraclass correlation coefficient (2,1) for the remaining expert-rated metrics. Between-model comparisons used paired nonparametric methods, with Kendall's W and 95% confidence intervals reported as the omnibus effect-size measure for Friedman tests. RESULTS: A total of 160 model-question responses were evaluated. Inter-rater agreement was substantial to excellent across metrics, with Fleiss' kappa of 0.892 for safety and intraclass correlation coefficient values ranging from 0.768 to 0.886 for the remaining metrics. Most responses were classified as safe; observed safe-response proportions differed descriptively across models, and no pairwise safety comparison remained statistically significant after Benjamini-Hochberg adjustment. Accuracy and empathy differed significantly among models (both P < 0.001): ChatGPT achieved the highest median accuracy score. DISCERN, EQIP, and JAMA scores differed significantly among models, whereas Global Quality Scale scores did not. All readability indices differed significantly among models: Doubao had the highest grade-level readability scores and the lowest Flesch Reading Ease score, whereas Gemini had comparatively easier readability. CONCLUSIONS: LLMs produced generally safe and moderately high-quality responses to adenomyosis-related patient questions, but performance varied by model and by evaluation domain. These findings support the need for transparent, multidimensional assessment of LLM-generated health advice before patient-facing use.