Verification feedback and self-revision for omissions in clinical recommendations by large language models: A paired evaluation
Large language models (LLMs) in clinical decision support can omit required actions (omission) or recommend wrong ones (commission). Verification layers check responses, and models can revise without feedback. Based on 176 test cases, we delivered the first responses of three LLMs (GPT-6 Astra, Gemini 3.1 Pro, Claude Opus 5.5) in six versions: no feedback, the full findings of a rule-and-agent verification framework, or alerts calibrated to a 10% false-alert budget, each appended or returned to the model for revision. Three LLM judges scored the share of high-grade (high-certainty) items on LLM-generated checklists that each text omitted; physicians reviewed samples. Revision without feedback (self-revision) lowered omission from 8.1% to 7.1% (−0.9 percentage points; 95% confidence interval [CI]: −1.7 to −0.1). In post hoc comparisons, revision with the full findings omitted 2.2 percentage points more than self-revision (95% CI: 1.3 to 3.2), or 0.8 (95% CI: −0.1 to 1.8) without the Gemini 3.1 Pro revisions that replaced content with status phrases. Revision with the alerts, which reached 56 of 528 first responses, did not differ from self-revision (0.5 percentage points; 95% CI: −0.3 to 1.3). Appended, the full findings changed judged omission by −0.6 percentage points (95% CI: −1.20 to −0.04). Based on 354 responses to physician-reviewed commission probes, both deliveries of the full findings lowered commission, mostly citations to nonexistent guidelines, from 6.8% to 3.4% (appended) and 4.2% (revised); with hedged commission counted, only revision lowered it (post hoc). Revision with verification feedback did not reduce omission beyond self-revision, which evaluations should include as a baseline.
Authors
- Rajaie Batniji
- Parth Sheth (ORCID: https://orcid.org/0009-0008-8983-5835)
- Sanjay Basu (ORCID: https://orcid.org/0000-0002-0599-6332)
- John Morgan (ORCID: https://orcid.org/0000-0003-3418-7558)
- Sadiq Patel
Institutions
- University of California, San Francisco (US)
- Virginia Commonwealth University (US)
- University of Pennsylvania (US)
Publication Details
- Journal
- Artificial Intelligence in Health
- Published
- 2026-11-17
- DOI
- https://doi.org/10.36922/aih026370137
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00