Extended-reasoning mode and large language model accuracy on the Orthopaedic In-Training Examination: a paired comparison of two ChatGPT configurations

Abstract Background Orthopaedic surgery residents are increasingly use large language models (LLMs) for studying and board preparation. Extant data compare models without isolating the effects of inference-time reasoning depth within a single model on an identical item set. We aimed to determine whether extended-reasoning inference improves LLM accuracy on the Orthopaedic In-Training Examination (OITE) relative to standard-mode output from the same model, and to characterize the failure behavior relevant to residents using these tools. Methods Cross-sectional paired benchmarking study. The 203 publicly released items of the 275-item 2024 OITE, obtained through American Academy of Orthopaedic Surgeons (AAOS) ResStudy, were each administered twice to a single multimodal LLM (GPT-5; OpenAI, San Francisco, CA) through the native ChatGPT multimodal interface between November 2025 and January 2026, once in standard mode and once in extended-reasoning mode. Prompting conditions were held identical and scored against the official AAOS answer key. Paired accuracy was compared with McNemar’s exact test; subspecialty differences with the Fisher-Freeman-Halton exact test. Results Accuracy was 79% (161/203) in standard mode and 93% (189/203) in extended-reasoning mode. Every item answered correctly in standard mode was also correct in extended-reasoning mode; all 28 discordant pairs favored extended reasoning (McNemar’s exact p < 0.001; absolute improvement 13.8 percentage points, 95% CI 9.0–18.5). Subspecialty accuracy ranged from 50% to 100% (standard) and 83% to 100% (extended reasoning), without significant between-subspecialty differences (p = 0.11 and p = 0.68). Paired improvement was significant for hand (50% to 83%; p = 0.031) and trauma (78% to 92%; p = 0.016). Accuracy did not differ between image-containing and text-only items in either configuration (both p = 1.0). Answer–rationale discordance occurred in 2% of items; incorrect answers were delivered without expressed uncertainty, at times with citations. Conclusions Inference-time reasoning depth materially changes LLM accuracy on orthopaedic board-style questions and is a setting resident’s control. Confidently delivered errors persisted under extended reasoning, supporting use as a supervised adjunct rather than a primary study resource.