Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes

Objective: To evaluate the accuracy and calibration of Decision and System One models on rheumatology examination questions and diagnostic vignettes. Methods: Two benchmarks were analysed: 172 text-based rheumatology questions from Spanish specialty-training examinations (expanded RheumaMIR dataset) in Spanish and English, and 70 English diagnostic vignettes from RheumBench, each with 5-18 candidate diagnoses, 52 of which included the correct reference diagnosis. Four System One models (i.e., Jev 1.13, Clef, Microsoft-Decision-1, Laya) and GPT-6 Luna, evaluated as a decision model through the OpenAI Decisions API, were compared with GPT-6 Sol, a frontier generative model, and two local generative models (Gemma 4 12B, Qwen3-8B). The primary outcome was accuracy, the proportion of questions or cases in which the selected option matched the answer key; secondary outcomes included calibration, discrimination, accuracy of answers with probability ≥0.90 and, in RheumaMIR, language differences. Results: In RheumaMIR, Jev’s accuracy was 93.0% in Spanish and 91.9% in English, below GPT-6 Sol (99.4% in both languages; Holm-adjusted P≤0.005); GPT-6 Luna reached 97.7% in both languages, Clef 91.9% and 93.0%, and Microsoft-Decision-1 87.8% and 86.0%, respectively, without detectable language differences. Jev’s answers with probability ≥0.90 (76% and 77% of answers) were 98.5% correct. In RheumBench, accuracy was lower for every model: GPT-6 Sol 76.9%, GPT-6 Luna 63.5%, Clef 59.6%, Jev 50.0% and Microsoft-Decision-1 40.4% (each System One model below GPT-6 Sol, Holm-adjusted P≤0.045), against 9.6% expected from random guessing. Jev, whose probabilities matched its accuracy in RheumaMIR (over/underconfidence index −0.015 and 0.000), became overconfident (mean probability 0.659; index +0.159), and its answers with probability ≥0.90 (25% of cases) were 84.6% correct, against 95.5% for GPT-6 Luna (42% of cases). Laya performed near chance in both benchmarks. Conclusion: The best-performing System One models achieved high accuracy on rheumatology examination questions, but their performance declined on diagnostic vignettes and remained below GPT-6 Sol. Jev’s high-confidence predictions were less reliable on diagnostic vignettes than on examination questions, highlighting the need for task-specific evaluation and improved calibration of diagnostic probabilities.