Evaluation of the content and readability of Google and ChatGPT information on tuberculosis

Abstract Background Tuberculosis (TB) is an infectious disease caused by bacteria and constitutes a significant public health problem. With expanding Internet and AI use, individuals seek health information online. This study assessed the content and readability of TB-related information from Google and ChatGPT. Methods A 14-item questionnaire based on FAQs about TB (7 from WHO and 7 from CDC), grouped into ‘General Information (GI)’ and ‘Diagnosis and Treatment (DT),’ was prepared. Questions entered into ChatGPT 4o-mini and searched on Google on February 3, 2025. Two researchers independently evaluated the answers for comprehensiveness and accuracy using WHO and CDC booklets and a 5-point Likert scale. Inter-rater agreement was assessed with Cohen's Kappa. Readability was assessed with the Ateşman and Bezirci-Yılmaz formulas. Statistical analysis included the Mann-Whitney U test, independent samples t-test, and Spearman correlation analysis. Results FAQ accuracy scores did not differ significantly between groups. In the GI group, comprehensiveness scores for ChatGPT and Google were 4.4 ± 0.5 (range 4.0-5.0) and 2.3 ± 1.4 (1.0-4.5), respectively (p = 0.017). In the DT group, they were 4.1 ± 0.6 (3.0-5.0) and 2.2 ± 0.9 (1.0-3.5), respectively (p = 0.002). 13 ChatGPT and 2 Google answers scored ≥4 for comprehensiveness (p < 0.001). Mean Ateşman scores were 54.0 (ChatGPT) and 54.1 (Google), and Bezirci-Yılmaz scores were 16.1 and 14.9 (p > 0.05), with a moderate negative correlation between the readability measures (r = -0.598; p = 0.001). The total inter-rater agreement was Cohen's Kappa = 0.836. Conclusions In the FAQs, ChatGPT showed better comprehensiveness than Google, while both sources were found to have similar readability at the undergraduate level. In the infodemic era, improving the comprehensiveness and readability of online sources like Google and ChatGPT is essential for effective TB education, control and public health. Key messages • Health authorities should enhance comprehensiveness of online health information. • Increasing online health information readability would improve public understanding and health outcomes.