Diagnostic accuracy of ChatGPT o1 versus ChatGPT 4o and their mini versions on MedQA clinical vignettes: a comparative evaluation

ChatGPT o1 introduces “chain-of-thought reasoning” (CoT), a structured reasoning approach potentially relevant to the management of medical cases. This study aimed to assess the diagnostic accuracy of ChatGPT o1 compared with other model iterations on standardized multiple-choice clinical vignettes. A total of 350 general medicine clinical cases from the MedQA dataset were analyzed using ChatGPT o1, ChatGPT o1 mini, ChatGPT 4o, and ChatGPT 4o mini. The diagnostic accuracy of these models was evaluated by comparing the proportion of correct responses; pairwise comparisons were performed with the exact McNemar test for paired data. ChatGPT o1 demonstrated a diagnostic accuracy of 93.4%, significantly surpassing ChatGPT 4o (82.3%, p < 0.001) and ChatGPT o1 mini (70.3%, p < 0.001), while the difference between ChatGPT o1 mini (70.3%) and ChatGPT 4o mini (66.3%) was not statistically significant ( p = 0.21). These findings suggest that reasoning-oriented models may achieve higher accuracy on standardized multiple-choice clinical vignettes. However, our design did not directly quantify hallucinations, and the results should be interpreted with caution given the public availability of the MedQA dataset and its possible inclusion in the models’ training data.