Evaluating ChatGPT’s Accuracy and Helpfulness in Gastroesophageal Reflux Disease Queries Compared to Traditional Search Engines: A New Era of Health Information Retrieval

Abstract The use of Artificial Intelligence (AI) is exponentially growing across medical specialties, offering numerous advantages over traditional search engines. This manuscript aims to fill this gap by evaluating the performance of the AI system ChatGPT in the context of Gastroesophageal Reflux Disease (GERD) related queries compared to traditional search engine responses. We compared the AI system ChatGPT 3.5 with the traditional search engine Google. Six standardized queries on GERD topics (definition, risk factors, diagnosis, symptoms, and management) were generated using Google’s auto-completions. Responses from ChatGPT 3.5 and Google were evaluated for accuracy and helpfulness using a five-point Likert scale by three independent gastroenterology experts. Statistical analysis was performed using Python. For accuracy, ChatGPT received average ratings of 5, 4.67, and 4.67, while traditional search engines scored 4.83, 4.33, and 3.33 ( p -values 0.363, 0.105, 0.001). For helpfulness, ChatGPT was rated 5, 4.67, and 4.83, compared to 4.33, 4, and 3.33 for traditional search engines ( p -values 0.012, 0.025, 0.004). Evaluator 3 found ChatGPT significantly more accurate, while evaluators 1 and 2 saw no significant difference. All three evaluators rated ChatGPT as significantly more helpful. Overall, ChatGPT received higher ratings for both accuracy and helpfulness. This study demonstrates that AI systems like ChatGPT 3.5 provide more accurate and useful information on GERD than traditional search engines like Google. AI can enhance patient education and decision-making, though further research is needed to confirm its reliability across other topics.