Evaluation of the appropriateness and readability of chatgpt-4, claude, copilot and gemini responses to patient queries on eye health

Aim: This study aims to assess the appropriateness and readability of ChatGPT-4, Claude, Copilot, and Gemini's responses to frequently asked questions about eye health.Material and Methods: Four commonly used Large Language Model (LLM) chatbots were asked 30 questions frequently asked by patients in an ophthalmology clinic.The appropriateness of the answers was evaluated by 2 experienced ophthalmologists.Readability was evaluated with 6 different indexes.Results: The Likert scale (5-1) was utilized to determine adequacy.The responses generated by ChatGPT-4 had the highest average score of 4,90, followed by Anthropic Claude with 4,23, Google Gemini with 3,87, and last was Microsoft Copilot with a score of 3,50.None of the chatbot responses were rated as "very poor" or classified as incorrect.With an average Flesch-Kincaid Grade Level readability score of 7.60, only Anthropic Claude met the level of readability that The American Medical Association recommends.An assessment of the remaining five parameters revealed that the responses generated by all four chatbots exhibited readability levels exceeding that of a sixth-grade.Discussion: In this study, ChatGPT-4 was more precise in responses, while Anthropic Claude scored better in terms of readability.These findings suggest that while LLMs may offer more comprehensive information, their complexity can hinder the average patient's comprehension.