Reliability and readability of large language model chatbot responses to public-facing questions about hypertension, antihypertensive therapy, and erectile dysfunction: a cross-sectional comparative study

Abstract Background Large language model (LLM) chatbots are increasingly used as public-facing sources of health information. This study compared the reliability, information quality, source transparency, and readability of four publicly accessible LLM chatbots responding to questions about hypertension, antihypertensive therapy, and erectile dysfunction (ED). Methods In this cross-sectional web-based study, 37 questions were retained from a 99-item pool informed by MeSH/PubMed, Google Trends, and recurring themes on publicly accessible Chinese online platforms. On May 22, 2026, each question was submitted once in a new conversation to ChatGPT 5.5, Microsoft Copilot, Gemini 3.5 Flash, and Perplexity, yielding 148 responses. Two blinded clinicians independently rated responses using DISCERN, EQIP, JAMA, and GQS; readability was assessed with six indices. Results Inter-rater agreement was excellent for DISCERN and EQIP (ICC(2,1) = 0.982 and 0.991) and almost perfect for JAMA and GQS (quadratic-weighted kappa = 0.851 and 0.92). Perplexity had the highest median DISCERN (47.00), EQIP (75.00), and JAMA (1.00) scores, with significant between-model differences (all P < 0.001). GQS did not differ significantly (P = 0.055; all medians 4.00). No model met the prespecified sixth-grade readability target; Copilot was descriptively easiest to read, but its median FKGL was 11.65 and FRES 38.60. Conclusion In this single-date benchmark, Perplexity achieved higher structured quality and transparency scores, whereas readability remained poor across all models. The findings support cautious use of general-purpose LLMs as supplementary rather than standalone patient-education sources and highlight the need for clearer source attribution and plain-language optimization.