Bilingual and multimodal evaluation of large language models and oral and maxillofacial surgeons for differentiating oral leukoplakia from oral squamous cell carcinoma: a retrospective single-centre disease-enriched diagnostic reader study

Differentiating oral leukoplakia from oral squamous cell carcinoma (OSCC) is clinically important; however, visual assessment alone can be challenging. This study compared two multimodal large language model (LLM) configurations with two oral and maxillofacial surgeons in a forced binary classification task and examined the effects of histopathological input, prompt language, session protocol, repeatability, and the behaviour of model-reported probability estimates. This retrospective, single-centre, disease-enriched diagnostic reader study was reported in accordance with the STARD 2015 and STARD-AI reporting guidelines. The dataset was deliberately balanced by selecting 50 cases from each diagnostic category (50 oral leukoplakia and 50 OSCC; 100 patients in total); it therefore does not reflect clinical prevalence. Clinical photographs and histopathological regions of interest were retrospectively selected by investigators who were aware of the final clinicopathological diagnosis. Two oral and maxillofacial surgeons and two multimodal LLM configurations (GPT-5.5 Thinking and Gemini 3 Pro) independently classified each case as oral leukoplakia or OSCC under two input conditions: clinical-only information and clinical information supplemented with histopathological images. The final clinicopathological diagnosis issued by the institutional pathology service was used as the reference standard; the histological images presented to evaluators originated from the same specimens used to establish that reference diagnosis. Diagnostic performance, evaluator agreement, OSCC classification tendency, language and session effects, repeatability across five independent runs, and model-reported OSCC probability estimates were analysed. In the clinical-only condition, diagnostic accuracy ranged from 88.0% to 94.0% across evaluators. GPT-5.5 Thinking and Gemini 3 Pro achieved accuracies of 93.0% and 94.0%, respectively, and no statistically significant differences between evaluators were detected after Holm adjustment; the study was not designed as an equivalence or non-inferiority investigation, and these results therefore do not establish equivalent performance. Following the addition of histopathological images, accuracy increased to 99.0%–100.0% across evaluators, with an overall posterior median improvement of + 6.6% points (95% credible interval, + 4.0 to + 10.2). Korean–English prompt differences were small, ranging from 0 to 2% points. In an exploratory single-run comparison, sequential-session evaluation reduced clinical-only accuracy from 93.0% to 78.0% for GPT-5.5 Thinking and from 94.0% to 82.0% for Gemini 3 Pro; because this comparison was based on one run per model, it cannot be distinguished from ordinary stochastic variation and is reported as hypothesis-generating. Across repeated independent runs, case-level stability ranged from 92.0% to 99.0%. Model-reported OSCC probabilities showed high discrimination in the clinical-only condition, whereas calibration could not be reliably assessed in the clinical-plus-histology condition because of near-perfect or perfect separation. Within this selected, balanced, single-centre dataset, no statistically significant difference in diagnostic accuracy was detected between the two multimodal LLM configurations and the two oral and maxillofacial surgeons. These findings should not be interpreted as evidence of equivalence or non-inferiority. The near-perfect accuracy observed after histopathological input was obtained in a reader task using representative fields selected from already diagnosed specimens and does not demonstrate general histopathological competence. Session-protocol sensitivity, diagnosis-aware image selection, the restricted diagnostic spectrum, and the limited human comparator group indicate that these results are preliminary and do not support clinical implementation before prospective, externally validated confirmation.