Performance of chatbots for clinical decision-making in orthodontics and dentofacial orthopedics

Objectives: Artificial intelligence utilization in medical fields has been expanded in recent years. The aim of this study was to compare the performance of three chatbots in answering clinical questions raised from an orthodontic and orthopedic clinical practice guideline (CPG). Material and Methods: Two orthodontists generated 60 questions from the CPG of the American Association of Orthodontics: 20 true-false questions, 20 multichoice questions, and 20 long-answer questions. Five researchers asked each question to ChatGPT-4, Copilot, and Gemini. The answers to true-false and multichoice questions were graded as wrong or right, and accuracy was analyzed using one-way analysis of variance (ANOVA). Answers to long-answer questions were graded from 0 to 10 in four categories (comprehensiveness, scientific accuracy, clarity, and relevance) and analyzed by Kruskal– Wallis test. A p < 0.05 was assumed statistically significant. Results: The overall accuracy of the three chatbots in answering true-false and multichoice questions was 0.7–0.9 and 0.5–0.8, respectively. There was no significant difference between the three chatbots in answering true-false questions, while multichoice questions were answered differently ( p < 0.035), with Copilot being slightly more accurate. For long answer questions, median grades were 8–9 for comprehensiveness, 10 for scientific accuracy, 10 for clarity, and 10 for relevance. Only for relevance did we find statistically significant differences between the chatbots ( p < 0.029). Conclusion: All three chatbots have substantial potential in answering orthodontic clinical questions, while performance varied greatly depending on question type.