Safety, accuracy, empathy and information quality of six large language models answering patient-facing questions about diabetic kidney disease: a cross-sectional comparative study

Background Large language models (LLMs) may support patient education, but their suitability for diabetic kidney disease (DKD) remains uncertain. We compared six publicly accessible LLM chatbots—ChatGPT, Gemini, DeepSeek, Doubao, Kimi, and Qwen—across safety, accuracy, empathy, information quality, and readability, with safety as the primary outcome. Methods In this CHART-guided cross-sectional study, 42 standardized English DKD questions were submitted once to each chatbot in independent single-turn sessions, generating 252 responses. Five source-masked clinical raters assessed safety, accuracy, empathy, DISCERN, EQIP, JAMA benchmarks, and the Global Quality Scale; readability was assessed automatically. Agreement was evaluated using Fleiss' κ and ICC (2,1)/ICC (2,5). Model differences were tested using Cochran's Q or Friedman tests with Kendall's W and Benjamini–Hochberg correction. Potentially unsafe responses underwent qualitative review. Results Safety showed substantial agreement ( κ = 0.854); ICC (2,1) ranged from 0.552 to 0.855 and ICC (2,5) from 0.860 to 0.967. Safe-response rates ranged from 83.3% to 92.9% without significant between-model differences ( P = 0.726). Accuracy and empathy differed significantly (both P < 0.001), with ChatGPT highest for accuracy and Gemini highest for clinician-rated empathy. Information quality also differed significantly, with ChatGPT performing best overall and Doubao lowest. All six readability indices differed across models (all P < 0.001; Kendall's W = 0.139–0.255). DeepSeek had the lowest mean scores on all five grade-level indices and the highest mean FRES, but mean grade-level estimates exceeded the sixth-grade reference level for every model. Thirty of 252 responses (11.9%) were potentially unsafe, most often because of incomplete medication-related information or insufficiently individualized fluid advice. Conclusions Publicly accessible LLMs can provide plausible DKD information but show variable performance and safety risks. They may support, but should not replace, professional patient education; guideline anchoring, source verification, improved readability, and clinical oversight remain essential.