Clinical reliability and consistency of large language models in prosthodontics: an expert evaluation of ChatGPT and Gemini

Abstract Background: The integration of artificial intelligence in medicine, both for education and clinical practice, has become widespread. In dentistry, these tools may be beneficial in terms of learning, clinical practice, and access to information. Nevertheless, their accuracy in specialized fields of dental medicine such as prosthodontics remains insufficiently explored. This study aims to evaluate the accuracy of the responses generated by ChatGPT and Gemini in prosthodontics, compare their performances in removable partial prosthodontics, complete removable prosthodontics, fixed prosthodontics, occlusodontics, and maxillofacial prosthodontics. Moreover, it aims at analyzing the precision, consistency and reproducibility of the generated responses while identifying the main limitations observed during the use of these models. Methods: Seventy questions covering several fields of prosthodontics were developed based on recognized scientific recommendations and references. The questions were asked in French to ChatGPT (GPT-4.0) and Gemini Pro 1.5. Each question generated 30 responses per model. A total of 4,200 responses were analyzed. The evaluation was performed by three examiners using a five-point Likert scale assessing the factual accuracy of the responses. Results: Gemini demonstrated overall superior performance compared with ChatGPT in the majority of the studied subfields. In maxillofacial prosthodontics, 50% of the questions achieved more than 90% correct responses with Gemini compared to 21.48% with ChatGPT. Similar findings were observed in complete removable prosthodontics, removable partial prosthodontics, and fixed prosthodontics. In occlusodontics, the performances of both models were more comparable. Gemini showed better response stability with a concordance coefficient of κ = 0.62 compared with κ = 0.46 for ChatGPT. Although Gemini achieved a slightly higher mean accuracy, the difference was not statistically significant (p = 0.109). Both models showed difficulties with specialized terminology and complex concepts, with the occasional presence of hallucinations and erroneous or incomplete responses. Conclusion: Conversational artificial intelligence models show promising potential as complementary tools for learning and access to information in prosthodontics. However, their level of accuracy remains insufficient for autonomous use in such a specialized field. Gemini appears to provide more stable and consistent responses than ChatGPT, without however guaranteeing constant scientific accuracy. The development of models specifically trained in dentistry, as well as regular evaluations of their performance, appears necessary to improve their reliability and integration into education and clinical practice.