Beyond Accuracy: A Critical Review of Evaluation, Clinical Validation, and Safety Frameworks for Agentic AI in Medicine

Agentic artificial intelligence (AI) extends conventional predictive and generative AI by enabling systems to pursue goals through reasoning, planning, information retrieval, memory, tool use, and interaction with clinical workflows. These capabilities create new opportunities for medicine but also a distinct evaluation problem: the reliability of an agent cannot be inferred from the accuracy of its underlying model or final answer alone. Errors may arise during information retrieval, tool use, state updates, inter-agent communication, or autonomous action and may propagate across a clinical workflow. This critical review examines approaches for evaluating and clinically validating agentic AI in medicine. Existing frameworks, including TRIPOD+AI, CONSORT-AI, SPIRIT-AI, DECIDE-AI, STARD-AI, FUTURE-AI, and Good Machine Learning Practice, provide important foundations but were not primarily designed for systems with persistent state, repeated actions, tool use, and multi-agent interaction. Emerging agent benchmarks increasingly address interactive tasks, tool use, trajectory reliability, and adversarial testing. We propose a five-level evaluation framework spanning model, agent, trajectory, system, and clinical outcome, together with a clinical validation ladder progressing from component testing and simulation to retrospective evaluation, prospective silent deployment, supervised clinical use, bounded action, and post-deployment surveillance. Key gaps include standardized trajectory metrics, evaluation of correlated multi-agent errors, autonomy-aware validation, human–agent performance, and lifecycle monitoring. Radiotherapy is used as a high-stakes clinical exemplar. We argue that agentic AI should be evaluated as a dynamic sociotechnical system and that clinical evidence must extend beyond accuracy toward behavioral reliability, safety, and demonstrated clinical benefit.