Small Clinical Change, Big AI Decision? Metamorphic Testing of Large Language Models in Dental Clinical Decision-Making
Large language model (LLM) chatbots are increasingly consulted for dental decisions, but case-by-case accuracy does not show whether their advice changes when, and only when, a clinically decisive fact changes. Metamorphic testing checks whether answers to two nearly identical cases relate as clinical rules require, without defining a single correct answer for each case. We applied it to oral and maxillofacial surgery and emergency dental decision-making. Thirty guideline-anchored relations were locked before querying, and 60 synthetic case pairs differing in one variable were submitted with a structured prompt to the ChatGPT (GPT-6 Astra) and Gemini (Gemini 3.8) web interfaces. Two blinded specialists scored each pair, with blinded adjudication of disagreements. In the main analysis (58 pairs; two defective vignettes withdrawn), relations were violated in 0 ChatGPT and 8 Gemini pairs (13.8%; p = 0.016, accounting for clustering within relations); with all 60 pairs, 1 vs. 8 (p = 0.063). Gemini violations clustered at numerical thresholds, escalation red flags and a sex-label invariance relation, and 13 of 116 Gemini responses, but no ChatGPT response, contained a safety violation. Chatbots that appear competent case by case can fail at clinically decisive thresholds; metamorphic testing offers a reproducible audit before clinical use.
Authors
- Neşet Akay (ORCID: https://orcid.org/0000-0002-9921-4070)
- Hüseyin Yanık (ORCID: https://orcid.org/0000-0002-4386-5536)
- Fethi Atıl (ORCID: https://orcid.org/0000-0002-8286-4819)
Institutions
- Mersin Üniversitesi (TR)
Publication Details
- Journal
- Applied Sciences
- Published
- 2026-11-20
- DOI
- https://doi.org/10.3390/app162211505
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00