Effect of guideline-assisted prompting on large language model performance in medication-related osteonecrosis of the jaw–related endodontic decision-making

Clinical decision-making in patients receiving antiresorptive medications requires balancing tooth preservation against the risk of medication-related osteonecrosis of the jaw (MRONJ). This study compared the performance of ChatGPT, Claude, and DeepSeek in MRONJ-related endodontic scenarios and evaluated the effect of guideline-assisted prompting using the 2022 American Association of Oral and Maxillofacial Surgeons Position Paper. Eighteen fictional and standardized MRONJ-related endodontic clinical scenarios were developed based on the 2022 AAOMS Position Paper and current scientific evidence. Three contemporary LLMs (ChatGPT, Claude, and DeepSeek) were evaluated using a two-phase study design consisting of baseline and guideline-assisted assessments. In the second phase, the complete AAOMS Position Paper was provided before evaluation of the same scenarios. Two endodontic specialists independently scored the anonymized responses across five clinical decision-making domains using a six-point ordinal scale (0–5). The primary outcome was the mean of the five domain scores, yielding an overall clinical decision-making score ranging from 0 to 5. Inter-rater agreement was assessed using the initial ratings, and disagreements were resolved through consensus before analysis. Linear mixed models (LMMs) were used for the primary analyses, with cumulative link mixed models (CLMMs) performed as sensitivity analyses for the ordinal domain scores. Bonferroni-adjusted post hoc comparisons were performed where appropriate. At baseline, ChatGPT achieved a significantly higher mean overall score than Claude (4.68 ± 0.14 vs. 4.29 ± 0.38). Following guideline-assisted prompting, Claude’s mean score increased to 4.68 ± 0.28 (+ 0.39 points on the 0–5 scale), and the statistically significant difference between ChatGPT and Claude was no longer observed. DeepSeek consistently received lower overall scores than both models. A significant overall phase effect ( p < 0.001) and model-by-phase interaction ( p = 0.004) were identified. Domain-level CLMM sensitivity analyses generally supported the LMM findings, although the phase effect for consultation and medication management was not confirmed by the CLMM. The most pronounced model-specific increase in guideline-concordant scores was observed for follow-up recommendations, particularly for Claude. Guideline-assisted prompting was associated with higher concordance with the predefined reference standard, particularly for Claude, which showed the largest improvement in overall scores and follow-up recommendations. Although ChatGPT scored significantly higher than Claude at baseline, this difference was no longer statistically significant following guideline-assisted prompting. Because the clinical scenarios, reference standard, and guideline-assisted intervention were aligned with the same AAOMS guideline, these findings should not be interpreted as independent validation of clinical accuracy. Further studies using externally developed scenarios and real-world clinical cases are required before considering these models for clinician-supervised decision support.