Small Clinical Change, Big AI Decision? Metamorphic Testing of Large Language Models in Dental Clinical Decision-Making

Large language model (LLM) chatbots are increasingly consulted for dental decisions, but case-by-case accuracy does not show whether their advice changes when, and only when, a clinically decisive fact changes. Metamorphic testing checks whether answers to two nearly identical cases relate as clinical rules require, without defining a single correct answer for each case. We applied it to oral and maxillofacial surgery and emergency dental decision-making. Thirty guideline-anchored relations were locked before querying, and 60 synthetic case pairs differing in one variable were submitted with a structured prompt to the ChatGPT (GPT-6 Astra) and Gemini (Gemini 3.8) web interfaces. Two blinded specialists scored each pair, with blinded adjudication of disagreements. In the main analysis (58 pairs; two defective vignettes withdrawn), relations were violated in 0 ChatGPT and 8 Gemini pairs (13.8%; p = 0.016, accounting for clustering within relations); with all 60 pairs, 1 vs. 8 (p = 0.063). Gemini violations clustered at numerical thresholds, escalation red flags and a sex-label invariance relation, and 13 of 116 Gemini responses, but no ChatGPT response, contained a safety violation. Chatbots that appear competent case by case can fail at clinically decisive thresholds; metamorphic testing offers a reproducible audit before clinical use.

Authors

Institutions

Publication Details

Journal
Applied Sciences
Published
2026-11-20
DOI
https://doi.org/10.3390/app162211505
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Small Clinical Change, Big AI Decision? Metamorphic Testing of Large Language Models in Dental Clinical Decision-Making

Neşet Akay, Hüseyin Yanık, Fethi Atıl
7 links
Applied Sciences
Artificial Intelligence in Healthcare and Education
article

Small Clinical Change, Big AI Decision? Metamorphic Testing of Large Language Models in Dental Clinical Decision-Making

Neşet Akay, Hüseyin Yanık, Fethi Atıl
article en

Abstract

Large language model (LLM) chatbots are increasingly consulted for dental decisions, but case-by-case accuracy does not show whether their advice changes when, and only when, a clinically decisive fact changes. Metamorphic testing checks whether answers to two nearly identical cases relate as clinical rules require, without defining a single correct answer for each case. We applied it to oral and maxillofacial surgery and emergency dental decision-making. Thirty guideline-anchored relations were locked before querying, and 60 synthetic case pairs differing in one variable were submitted with a structured prompt to the ChatGPT (GPT-6 Astra) and Gemini (Gemini 3.8) web interfaces. Two blinded specialists scored each pair, with blinded adjudication of disagreements. In the main analysis (58 pairs; two defective vignettes withdrawn), relations were violated in 0 ChatGPT and 8 Gemini pairs (13.8%; p = 0.016, accounting for clustering within relations); with all 60 pairs, 1 vs. 8 (p = 0.063). Gemini violations clustered at numerical thresholds, escalation red flags and a sex-label invariance relation, and 13 of 116 Gemini responses, but no ChatGPT response, contained a safety violation. Chatbots that appear competent case by case can fail at clinically decisive thresholds; metamorphic testing offers a reproducible audit before clinical use.

Applied SciencesVol. 16(22)
Mersin Üniversitesi (TR)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon