Comparative evaluation of ChatGPT, DeepSeek, and Doubao responses to patient questions on pulmonary contusion: quality, reliability, readability, and content deficiencies

Objective To compare the quality, reliability, understandability, actionability, readability, and content deficiencies of responses generated by ChatGPT, DeepSeek, and Doubao to common patient questions about pulmonary contusion. Methods Twenty frequently asked questions about pulmonary contusion were identified from Google Trends, common clinical consultations, and relevant clinical guidelines. Each question was submitted separately to ChatGPT, DeepSeek, and Doubao, producing 60 responses. Information quality and reliability were assessed using the GQS, mDISCERN, QAMAI, EQIP, and PEMAT (understandability and actionability). Readability was evaluated with six indices (ARI, FRES, GFI, FKGL, CLI and SMOG). Content deficiencies were classified as omissions, hallucinations, output errors, or outdated medical advice. Results Significant differences were found among the three models across all quality and reliability measures (Friedman test, all P < 0.05). ChatGPT had the highest median GQS (4.50), QAMAI (26.00), and EQIP (75.00%) scores, whereas DeepSeek had the highest mDISCERN score (18.00). ChatGPT also achieved the highest PEMAT-U score (median 83.50%), and its median PEMAT-A score (80.00%) was equal to that of DeepSeek. Pairwise comparisons showed that ChatGPT and DeepSeek did not differ significantly on GQS, EQIP, PEMAT-U, or PEMAT-A, but both significantly outperformed Doubao (all P _adj < 0.05). None of the models met the recommended sixth-grade reading level for public health information. ChatGPT produced the most difficult-to-read responses (median FKGL 16.85). A total of 44 content deficiencies were identified, most commonly omissions (52.27%), followed by hallucinations (25.00%) and outdated medical advice (13.64%). Quality-related measures were generally positively correlated, whereas associations between information quality and readability were weak. Conclusion ChatGPT and DeepSeek generally performed better than Doubao in providing information about pulmonary contusion, although each model showed different strengths. However, none of the models met the recommended readability standards for public health information, and clinically relevant content deficiencies were observed across all three. LLM-generated medical information should be used as a supplementary source rather than a substitute for professional medical advice.