Evaluating the Ethical and Clinical Reliability of Large Language Models in Oral and Maxillofacial Surgery
Background Large language models (LLMs) are becoming a subject of great interest as a clinical decision-support tool, but their reliability is not well described in the surgical field of oral and maxillofacial surgery (OMFS). Objective This study aim ed to assess three popular general- purpose LLMs in the context of clinical accuracy, ethical safety, and risk awareness in 10 high-stakes OMFS scenarios. Methods Standardized clinical scenarios for postoperative emergencies, implant planning, trauma, oncology, and therapeutic management were used to prompt GPT-4, Claude 3, and Gemini. The responses were evaluated independently by two OMFS board-certified surgeons on an ordinal scale of 0–2 in three areas: clinical accuracy, ethical safety, and risk awareness. Mean scores were calculated and non-parametric comparisons were made. Results All models demonstrated high clinical accuracy (mean ± SD: GPT-4: 1.85 ± 0.42; Claude 3: 1.90 ± 0.38; Gemini: 1.75 ± 0.48), but exhibited significant variability in ethical safety (Claude 3: 1.75 ± 0.45; GPT-4: 1.60 ± 0.58; Gemini: 1.50 ± 0.52; Kruskal-Wallis p = 0.004). The risk awareness scores ranged between moderate (1.65–1.80) across models. The post hoc tests showed that the ethical safety scores differed significantly between Claude 3 and Gemini (Dunn's test, p = 0.008). Conclusions Although LLMs can provide guideline-concordant recommendations, inconsistent ethical framing and lack of escalation process directions restrict safe implementation of LLM in OMFS. A human in the loop structure and the ethics tuning pertaining to the domain are necessary prior to clinical integration. Clinical Relevance: While LLMs can offer prompt, clinically appropriate information in concert with guidelines, there is a great risk of overlooking the necessity to escalate to a specialist case in OMFS. These tools should be used as a complementary support tool in a well-defined framework of human-in-the-loop, until domain-specific ethical tuning and real-world validation are attained. Thus, clinical decision-making and patient safety are well protected under the authority of a qualified surgeon.
Authors
- Omid Panahi
Institutions
- Yeditepe University (TR)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-19
- DOI
- https://doi.org/10.21203/rs.3.rs-11581048/v1
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- preprint