Verification feedback and self-revision for omissions in clinical recommendations by large language models: A paired evaluation

Large language models (LLMs) in clinical decision support can omit required actions (omission) or recommend wrong ones (commission). Verification layers check responses, and models can revise without feedback. Based on 176 test cases, we delivered the first responses of three LLMs (GPT-6 Astra, Gemini 3.1 Pro, Claude Opus 5.5) in six versions: no feedback, the full findings of a rule-and-agent verification framework, or alerts calibrated to a 10% false-alert budget, each appended or returned to the model for revision. Three LLM judges scored the share of high-grade (high-certainty) items on LLM-generated checklists that each text omitted; physicians reviewed samples. Revision without feedback (self-revision) lowered omission from 8.1% to 7.1% (−0.9 percentage points; 95% confidence interval [CI]: −1.7 to −0.1). In post hoc comparisons, revision with the full findings omitted 2.2 percentage points more than self-revision (95% CI: 1.3 to 3.2), or 0.8 (95% CI: −0.1 to 1.8) without the Gemini 3.1 Pro revisions that replaced content with status phrases. Revision with the alerts, which reached 56 of 528 first responses, did not differ from self-revision (0.5 percentage points; 95% CI: −0.3 to 1.3). Appended, the full findings changed judged omission by −0.6 percentage points (95% CI: −1.20 to −0.04). Based on 354 responses to physician-reviewed commission probes, both deliveries of the full findings lowered commission, mostly citations to nonexistent guidelines, from 6.8% to 3.4% (appended) and 4.2% (revised); with hedged commission counted, only revision lowered it (post hoc). Revision with verification feedback did not reduce omission beyond self-revision, which evaluations should include as a baseline.

Authors

Institutions

Publication Details

Journal
Artificial Intelligence in Health
Published
2026-11-17
DOI
https://doi.org/10.36922/aih026370137
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Verification feedback and self-revision for omissions in clinical recommendations by large language models: A paired evaluation

Rajaie Batniji, Parth Sheth, Sanjay Basu, John Morgan et al.
2 links
Artificial Intelligence in Health
Artificial Intelligence in Healthcare and Education
article

Verification feedback and self-revision for omissions in clinical recommendations by large language models: A paired evaluation

Rajaie Batniji, Parth Sheth, Sanjay Basu, John Morgan, Sadiq Patel
article en

Abstract

Large language models (LLMs) in clinical decision support can omit required actions (omission) or recommend wrong ones (commission). Verification layers check responses, and models can revise without feedback. Based on 176 test cases, we delivered the first responses of three LLMs (GPT-6 Astra, Gemini 3.1 Pro, Claude Opus 5.5) in six versions: no feedback, the full findings of a rule-and-agent verification framework, or alerts calibrated to a 10% false-alert budget, each appended or returned to the model for revision. Three LLM judges scored the share of high-grade (high-certainty) items on LLM-generated checklists that each text omitted; physicians reviewed samples. Revision without feedback (self-revision) lowered omission from 8.1% to 7.1% (−0.9 percentage points; 95% confidence interval [CI]: −1.7 to −0.1). In post hoc comparisons, revision with the full findings omitted 2.2 percentage points more than self-revision (95% CI: 1.3 to 3.2), or 0.8 (95% CI: −0.1 to 1.8) without the Gemini 3.1 Pro revisions that replaced content with status phrases. Revision with the alerts, which reached 56 of 528 first responses, did not differ from self-revision (0.5 percentage points; 95% CI: −0.3 to 1.3). Appended, the full findings changed judged omission by −0.6 percentage points (95% CI: −1.20 to −0.04). Based on 354 responses to physician-reviewed commission probes, both deliveries of the full findings lowered commission, mostly citations to nonexistent guidelines, from 6.8% to 3.4% (appended) and 4.2% (revised); with hedged commission counted, only revision lowered it (post hoc). Revision with verification feedback did not reduce omission beyond self-revision, which evaluations should include as a baseline.

Artificial Intelligence in HealthVol. 0(0)
University of California, San Francisco (US), Virginia Commonwealth University (US), University of Pennsylvania (US)
Good health and well-being
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon