The Evaluation Of Responses Provided By ChatGPT 4.0 to Different Educational Levels Regarding Orthognathic Surgery

Background: This study aims to evaluate ChatGPT's effectiveness in providing accurate and level-appropriate information on orthognathic surgery across different educational levels. As AI becomes more prevalent in healthcare, understanding its potential to support patient education and professional communication is crucial. The focus is on assessing the model's performance in addressing questions about postoperative care, surgical risks, and treatment procedures, and its implications for minimizing misinformation risks, such as AI hallucinations. Methods: ChatGPT 4.0 was used to answer frequently asked questions about orthognathic surgery, categorized by educational level. Questions were refined by researchers and responses were simulated for bachelor's and academic levels. The responses were then evaluated by two bachelor’s graduates and two academics. Cosine similarity was also calculated to compare the responses across different educational levels. Results: ChatGPT 4.0 provided highly accurate responses for both educational levels, with average accuracy scores of 4.58±0.7980 for bachelor's and 4.4107±0.8745 for academic levels (p=0,123). However, the appropriateness of responses varied, with bachelor's graduates rating the responses at 4.64±0.10 and academics at 3.05±0.319. Statistical tests confirmed a significant difference between the groups (p