Retrospective testing and prospective randomized trial of guardrails and a verifier for AI care plans
Health systems are adding verifiers, AI models that check another AI’s drafted care plans, to reduce errors and omissions. We tested a verifier in two ways. First, three AI models drafted care plans for 447 retrospective test cases; a verifier caught 5 of 17 errors checkable against the patient’s record when counting flags for wrong recommendations, and 10 of 17 when also including flags for missing care. Second, in a prospective trial (NCT07718893), we randomized 240 patients in complex primary care management programs in four states to unassisted physicians, physicians with a guardrailed AI assistant, or physicians with the guardrailed assistant plus a verifier. The verifier raised 11 alerts across 77 encounters; physician adjudicators judged 9 correct, but clinicians in the trial acted on only 1. Safety events were similar across arms. Effective verification may therefore require alerts that clinicians must resolve, despite the risk of alert fatigue.
Authors
- Rajaie Batniji
- Parth Sheth (ORCID: https://orcid.org/0009-0008-8983-5835)
- Sanjay Basu (ORCID: https://orcid.org/0000-0002-0599-6332)
- Aakriti Kinra
- Dan Wanek (ORCID: https://orcid.org/0009-0009-7670-2854)
- John Morgan
- Bernardo Arevalo
- Namrata Elemaran
Institutions
- University of California, San Francisco (US)
- Virginia Commonwealth University (US)
- University of Pennsylvania (US)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-18
- DOI
- https://doi.org/10.21203/rs.3.rs-11515604/v1
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- preprint