Artificial Intelligence Chatbots as Sources of Dyspepsia Information: An Evaluation of Quality and Readability

Abstract Background Large language models (LLMs) have made artificial intelligence chatbots an increasingly common source of medical and health information for the public. Patients with dyspepsia seek information on disease definitions, symptom recognition, diagnostic evaluation, treatment choices, and subsequent management, yet the quality, source transparency, and readability of content generated by different AI chatbots have not been systematically evaluated. Objective To compare the quality, transparency, and readability of dyspepsia-related health information generated by four mainstream AI chatbots (ChatGPT, Gemini, Copilot, and Perplexity) and assess whether their responses meet the recommended reading level for patient education materials. Methods A web-based cross-sectional comparative evaluation design was used. Using the MeSH term “Dyspepsia,” Google Trends was searched worldwide from August 31, 2021, to August 31, 2026. After screening, 25 core questions were entered into the four AI chatbots using the same original English wording. Information quality, patient education value, source transparency, and overall quality were assessed with DISCERN, Ensuring Quality Information for Patients (EQIP), the JAMA Benchmark Criteria, and the Global Quality Scale (GQS). Readability was assessed with the Automated Readability Index (ARI), Flesch Reading Ease Score (FRES), Gunning Fog Index (GFI), Flesch-Kincaid Grade Level (FKGL), Coleman-Liau Index (CLI), and Simple Measure of Gobbledygook (SMOG). Quality scores were compared across models using the Kruskal-Wallis test; significant overall tests were followed by Dunn post hoc pairwise comparisons with multiple-comparison correction, and ε² effect sizes were reported. Readability results were compared with the sixth-grade reading standard for patient education materials. Statistical analyses were performed using Python 3.10.4. Results The analysis included 25 core dyspepsia-related questions. DISCERN, EQIP, JAMA, and GQS scores differed significantly among the four AI chatbots, with Perplexity and Copilot performing relatively better on some measures of information quality and source transparency. None of the models met the prespecified sixth-grade reading standard for patient education materials. ARI, GFI, FKGL, CLI, and SMOG values exceeded the recommended thresholds for all models, whereas FRES values were below the recommended threshold, indicating high reading difficulty overall. Conclusion Mainstream AI chatbots rapidly generate structured dyspepsia-related health information, but information quality and source transparency differ across models, and inadequate readability is common to all. AI chatbots are currently better suited as supplementary tools for patients seeking health information. Broader medical use will require better source traceability, content quality, information currency, and linguistic comprehensibility.