Retrospective testing and prospective randomized trial of guardrails and a verifier for AI care plans

Health systems are adding verifiers, AI models that check another AI’s drafted care plans, to reduce errors and omissions. We tested a verifier in two ways. First, three AI models drafted care plans for 447 retrospective test cases; a verifier caught 5 of 17 errors checkable against the patient’s record when counting flags for wrong recommendations, and 10 of 17 when also including flags for missing care. Second, in a prospective trial (NCT07718893), we randomized 240 patients in complex primary care management programs in four states to unassisted physicians, physicians with a guardrailed AI assistant, or physicians with the guardrailed assistant plus a verifier. The verifier raised 11 alerts across 77 encounters; physician adjudicators judged 9 correct, but clinicians in the trial acted on only 1. Safety events were similar across arms. Effective verification may therefore require alerts that clinicians must resolve, despite the risk of alert fatigue.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-18
DOI
https://doi.org/10.21203/rs.3.rs-11515604/v1
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Retrospective testing and prospective randomized trial of guardrails and a verifier for AI care plans

Rajaie Batniji, Parth Sheth, Sanjay Basu, Aakriti Kinra et al.
2 links
Research Square
Artificial Intelligence in Healthcare and Education
preprint

Retrospective testing and prospective randomized trial of guardrails and a verifier for AI care plans

Rajaie Batniji, Parth Sheth, Sanjay Basu, Aakriti Kinra, Dan Wanek, John Morgan, Bernardo Arevalo, Namrata Elemaran
preprint en

Abstract

Health systems are adding verifiers, AI models that check another AI’s drafted care plans, to reduce errors and omissions. We tested a verifier in two ways. First, three AI models drafted care plans for 447 retrospective test cases; a verifier caught 5 of 17 errors checkable against the patient’s record when counting flags for wrong recommendations, and 10 of 17 when also including flags for missing care. Second, in a prospective trial (NCT07718893), we randomized 240 patients in complex primary care management programs in four states to unassisted physicians, physicians with a guardrailed AI assistant, or physicians with the guardrailed assistant plus a verifier. The verifier raised 11 alerts across 77 encounters; physician adjudicators judged 9 correct, but clinicians in the trial acted on only 1. Safety events were similar across arms. Effective verification may therefore require alerts that clinicians must resolve, despite the risk of alert fatigue.

Research Square
University of California, San Francisco (US), Virginia Commonwealth University (US), University of Pennsylvania (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon