Accuracy Is Not Enough: Reasoning and Reference Reliability in Orthopaedic Large Language Model (LLM) Applications
GPT-5 appears to exceed previously reported LLM performance on the OITE and achieved accuracy higher than published mean scores for senior trainees, but demonstrated poor reference reliability, with one in three answers citing fabricated or misrepresented evidence. Even correct answers frequently relied on flawed or unverifiable sources. Evaluation of LLMs in medical education should incorporate systematic reasoning and evidence validation, not accuracy alone.
Authors
- Pranav Chandrasekhar
- Shashwat Singh
Institutions
- University of East Anglia (GB)
- Queen Elizabeth Hospital King's Lynn NHS Foundation Trust (GB)
Publication Details
- Journal
- Cureus
- Published
- 2026-01-05
- DOI
- https://doi.org/10.7759/cureus.100845
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00