Large language model responses to postoperative questions in oral and maxillofacial surgery: a comparative evaluation

Abstract Background Large language models (LLMs) are increasingly used as patient-facing informational tools in dentistry, but their suitability for postoperative communication in oral and maxillofacial surgery (OMFS) remains uncertain. This study compared ChatGPT-5.4 Thinking, DeepSeek-V3.2, and Gemini 3 (Thinking mode) in answering routine postoperative OMFS questions. Methods Thirty standardised open-ended postoperative questions were submitted once to each model through public web interfaces using the same neutral prompt. Three OMFS surgeons, masked to model identity, independently rated overall informational quality using an adapted 5-point Global Quality Score (GQS). Factual accuracy, completeness, and clinical safety were rated against a question-specific Reference Answer and Safety Framework; all responses were reassessed after peer review, and both rating sets were analysed. Five readability indices and user-interface latency were recorded. Friedman tests with Bonferroni-adjusted Wilcoxon post hoc tests were used. Results GQS inter-rater reliability was excellent [ICC(2,1), 0.900–0.926]. Mean GQS differed across models ( p < 0.001): Gemini 3 (Thinking mode) 4.82 ± 0.38, DeepSeek-V3.2 4.53 ± 0.49, and ChatGPT-5.4 Thinking 4.08 ± 0.64. Original accuracy and completeness ratings showed limited agreement [ICC(2,1), 0.379 and 0.272]. Factual accuracy differed in the original ratings ( p = 0.014) but not after reassessment ( p = 0.115); completeness did not differ. Of 90 responses, 89 were classified as Safe; one Gemini response on clopidogrel resumption was classified as Missed Red Flag/False Reassurance. After adjustment for word count, Gemini retained higher GQS than both other models (both p < 0.001). Gemini responses were generally more linguistically complex. Observed latency was shortest for DeepSeek-V3.2 (10.49 ± 1.95 s) and longest for Gemini (31.46 ± 11.60 s) and does not represent intrinsic model speed. Conclusions Gemini 3 (Thinking mode) achieved the highest expert-rated informational quality, but between-model differences in factual accuracy were not robust to reassessment, completeness did not differ, and one clinically consequential safety concern was identified. The findings are limited to simulated, single-turn, English-language first responses and do not establish patient comprehension or real-world safety. LLM-generated postoperative information should remain under professional oversight.