How Well Do AI Models Inform Otitis Media Patients? A Comparative Study of Large Language Models

Aim: The aim of this study was to evaluate the accuracy, comprehensiveness, and readability of information provided by large language models (LLMs) supported by natural language processing (NLP) technologies regarding otitis media (OM) based on responses to questions asked by patients and their relatives. Method: In this descriptive, cross-sectional evaluation study, 60 frequently asked questions by patients and their relatives regarding OM were classified under four subheadings (general information, diagnosis, follow-up and therapy, and surgery and complications) and answered by three different LLMs (Google Gemini 2.5 Flash, Microsoft Copilot, ChatGPT-4o). The answers were evaluated for accuracy by two experienced otorhinolaryngology specialists using a 5-point Likert scale. The readability of the responses was analyzed using the Coleman-Liau Index (CLI) and Simple Measure of Gobbledygook (SMOG) index to determine readability levels corresponding to academic education levels and to compare the models. Results: The artificial intelligence (AI) models received similarly high scores for accuracy in their responses to patient questions related to OM. In the readability analysis, Gemini responses were found to be statistically significantly more readable than those of the other models, according to the SMOG and CLI indices. The ChatGPT responses required a higher level of education; in particular, the readability of the answers under the “diagnosis” subheading was found to have the highest rate of graduate-level education requirement. Conclusion: Although the three commonly used AI models provided similarly accurate responses to OM-related questions, differences in readability were observed among the LLMs. For AI to effectively support patient education and promote treatment adherence, both the accuracy and readability of the content are essential.