HealthBench: Advancing AI evaluation in healthcare, but not yet clinically ready

Large language models (LLMs) are increasingly applied in healthcare, yet their evaluation remains inconsistent and often disconnected from clinical practice. OpenAI's HealthBench represents an important advancement, encompassing 5000 multiturn synthetic clinical conversations benchmarked against 48,562 clinician-developed criteria across accuracy, completeness, context awareness, communication, and instruction-following. Key strengths include broad scenario coverage, contributions from 262 clinicians across 60 countries, and automated grading methods that show high concordance with physician ratings. HealthBench provides a scalable and globally relevant framework. Nevertheless, important limitations constrain its clinical applicability. Exclusive reliance on synthetic dialogs limits ecological validity, and model-based graders may reinforce shared blind spots. Moreover, HealthBench assesses static, offline interactions while omitting multimodal inputs, longitudinal care, and patient outcomes-factors critical to real-world decision-making. Without external validation, strong benchmark performance may not translate into improved diagnostic accuracy, workflow efficiency, or patient safety. To ensure safe and effective integration of LLMs into practice, future benchmarks must incorporate authentic clinical data, longitudinal outcomes, and system-level considerations. HealthBench is a valuable step, but evaluation strategies must evolve to capture the complexity and demands of frontline care.