Evaluating AI chatbots as sources of public-facing encephalitis health information: a cross-sectional comparative study of quality, transparency, and readability

Background Publicly accessible AI chatbots are increasingly used as interactive sources of health information, yet their suitability for patient-facing communication depends on information quality, source transparency, and readability. Encephalitis is a clinically heterogeneous and potentially serious condition for which patients and caregivers may seek information about symptoms, causes, diagnosis, imaging, and treatment. Objective To evaluate the quality, transparency, and readability of encephalitis health information generated by four publicly accessible AI chatbots and to consider the implications for patient-oriented digital health communication. Methods This web-based cross-sectional study evaluated 32 core encephalitis-related queries identified using Google Trends, each submitted once to four LLM chatbots: ChatGPT, Perplexity, Gemini, and Copilot. The responses were assessed using DISCERN, Ensuring Quality Information for Patients (EQIP), the JAMA benchmark criteria, and the Global Quality Scale (GQS), along with six validated readability indices. Differences among chatbots were analyzed using the Friedman test with paired post hoc comparisons and Holm correction. Results All 32 queries were analyzed. DISCERN, EQIP, and JAMA scores differed significantly among the four chatbots (all p < 0.001), whereas GQS scores did not ( p = 0.090). Perplexity and Copilot had higher median DISCERN and EQIP scores than ChatGPT and Gemini, but pairwise differences were not uniform across all measures. JAMA scores were higher for Perplexity and Copilot than for ChatGPT and Gemini in most pairwise comparisons. None of the four chatbots met the prespecified sixth-grade readability benchmarks. Conclusion In this single-time-point evaluation of 32 English-language encephalitis queries, chatbots differed across selected information-quality and transparency measures, while none met the prespecified sixth-grade readability benchmark. Further evaluation of response stability, factual accuracy, clinical safety, and patient-centered outcomes is needed to establish the implications for patient-oriented use.