EM-BENCH: A Reproducible Framework for Benchmarking Guideline-Concordant Clinical Decision Making in Large Language Models

Background Physicians increasingly use artificial intelligence tools to support clinical work, yet there is no accepted standard for measuring the competency of large language models (LLMs) in clinical decision support. We developed EM-BENCH, a physician-validated, deterministically scored benchmark for evaluating whether models reliably apply the American College of Emergency Physicians (ACEP) guidelines to clinical decisions. Methods EM-BENCH comprises 210 binary clinical vignettes covering all 70 critical questions across 25 ACEP clinical policies. We evaluated 149 LLM configurations representing 79 base models using prespecified answers and deterministic scoring. Each vignette had a prespecified correct answer to a binary yes-or-no question based on an ACEP recommendation. We examined associations between accuracy and model access tier (closed-access vs. open-weight models), model size, release date, clinical policy, recommendation evidence level, and reasoning settings. Results No configuration achieved complete guideline concordance. The highest performing configuration answered 207 of 210 vignettes correctly (98.6%). Closed-access models outperformed open-weight models (median accuracy, 91% vs. 63%; Hodges-Lehmann location shift, 26 percentage points; 99% CI, 20 to 33; adjusted P<0.001). Among open-weight model families, a 10-fold increase in parameter count was associated with a 14-percentage-point increase in accuracy (99% CI, 7.7 to 20; adjusted P<0.001). Across all models, accuracy increased by 8.9 percentage points per release year (99% CI, 3.6 to 14; adjusted P<0.001). Neither enabling a thinking mode (pseudomedian change, 0.95 percentage points; 99% CI, −2.6 to 3.1; adjusted P=1.00) nor high versus low reasoning effort (0.91 percentage points; 99% CI, −0.69 to 3.1; adjusted P=0.56) consistently improved accuracy. Conclusions Contemporary LLMs showed improving but incomplete ability to apply ACEP guidelines. Concordance varied markedly across models and was associated with access tier, size, and release recency, while additional inference-time reasoning offered inconsistent benefit. Specialty-specific, physician-validated benchmarks may provide a reproducible assessment of LLMs' ability to make guideline-concordant clinical decisions.