Evaluating the ability of artificial intelligence chatbots to respond to oral cancer questions: A descriptive study

Background: Individual diagnosed with cancer have excessive informational needs regarding various aspects of their disease. This need pushes them to search for information from multiple sources including Artificial Intelligence chatbots. This study aimed to evaluate the accuracy of responses of Artificial Intelligence chatbots to questions regarding different aspects of oral cancers. Methods: Nineteen dichotomous (yes/no) questions were posed to 8 different Artificial Intelligence Chatbots (Chat GTP 3.5, Gemini, Perplexity, Claude, Chat GTP alternative, TextCortex, YouChat, and CoPilot). The responses were recorded and compared with the correct answers. Statistical analyses, including independent sample t-tests, were conducted to compare differences in mean scores. Results: Most (n=14, 73.7%), questions were answered correctly, and only 1 (5.3%) question got two incorrect responses. Only 4 (50.0%) AI chatbots answered all questions perfectly, and only one chatbot scored less than 80% for all questions. The combined mean score for the AI chatbots was 18.22 ± 1.3, and the differences in the mean score of responses between the AI chatbots and blueprint were statistically insignificant (p= 0.562). Conclusion: The general accuracy of responses to different oral cancer-related questions raised by the patients was high. However, not all AI chatbots give correct responses to all questions, therefore it is the role of health professionals to ensure that patients are educated well regarding their diseases and cautioned on relying on AI chatbots totally.