Large Language Models in Israeli Medical Administration Board Exam: Performance, Calibration, and Implications for Health-System Governance

Abstract Background: Large language models (LLMs) are increasingly used in healthcare, but their reliability in medical administration remains poorly understood, particularly in areas involving local regulation, policy, and health-system governance. Methods: We evaluated six LLMs using 596 multiple-choice questions from Israeli Medical Administration Board examinations conducted between 2020 and 2025. Official resident benchmark data were available for 296 matched items. Outcomes included overall accuracy, comparison with resident performance, stratified performance by administrative domain, question modality, and examination year, and confidence-based discrimination and calibration metrics. Results: Overall accuracy varied across models, with Gemini-Pro-3 achieving the highest performance (77.9%; 95% CI, 74.3–81.1). In the matched subset, its accuracy was 78.0%, compared with a mean resident accuracy of 74.0%. More importantly, performance was significantly lower in domains requiring locally specific knowledge, including Israeli Ministry of Health Directives, Reports and Committees, and Legislation and Regulations, compared with Statistics and Epidemiology. Confidence reliability also varied across models: Gemini-Flash-3 showed the highest confidence-based discrimination, ChatGPT-5.4 had the lowest expected calibration error, and mean self-reported confidence exceeded observed accuracy in all models. Conclusions Contemporary LLMs showed variable performance across different areas of medical administration. Performance was weaker in locally specific regulatory and governance domains, and confidence did not consistently identify reliable responses. These findings support local validation, authoritative source verification, and human oversight when LLMs are used for health-system tasks.