Evaluation of the Quality and Readability of ChatGPT Responses to Toothache Queries: A Study Based on Google Trends

Background Conversational AI tools such as ChatGPT are increasingly used for health information seeking. While their popularity continues to grow, little is known about the readability and quality of their outputs in dental contexts, particularly for toothache, one of the most common oral health complaints. Objective This study aimed to evaluate the quality and readability of ChatGPT's responses to frequently searched toothache-related queries and to conceptually compare them with information available from top-ranked traditional websites. Methods In this cross-sectional study, the 20 most common toothache-related queries were identified using Google Trends (January 2014-January 2024). Each query was posed to ChatGPT (May 2024 version) in independent sessions to avoid contextual bias. Two endodontist raters assessed the quality of responses using the Ensuring Quality Information for Patients (EQIP) tool. Readability was measured using the Flesch Reading Ease, Flesch-Kincaid Grade Level, and Simple Measure of Gobbledygook (SMOG) Index. Interrater reliability was calculated using Cohen's kappa and the Intraclass Correlation Coefficient (ICC). For comparison, the first 3-5 non-advertising websites retrieved via Google for each query were evaluated using the same instruments. Results ChatGPT responses demonstrated high content quality (mean EQIP 85.3 ± 5.2) and moderate readability demands (Flesch Reading Ease 57.9 ± 3.2; Flesch-Kincaid 8.4 ± 0.6; SMOG 7.5 ± 0.4). Interrater reliability was excellent (κ = 0.86; ICC = 0.91). Compared with websites, ChatGPT yielded slightly higher EQIP scores (ΔEQIP + 3.1 on average) but also higher reading grade levels (ΔFKGL + 0.4). Conclusions ChatGPT provides responses to toothache-related queries that are comparable, and in some cases superior, in quality to top-ranked, non-advertising traditional health websites retrieved via Google search. However, both ChatGPT and these conventional sources exhibit moderate readability, which may limit accessibility for individuals with low health literacy. Future efforts should focus on simplifying AI-generated and online content to enhance clarity, equity, and effectiveness in digital health communication.