Helping a Weak AI Checker: More Catches, Many More False Alarms

Doctors now ask AI tools what to do about a patient, and they rarely have time to check each answer against the guidelines before acting on it. One safeguard is a second AI model, the checker, which reads the first one's answer and says whether it is safe. To save money and time, the checker is often a smaller and less capable model than the one it checks. It can be wrong in two ways: it can pass an unsafe answer (a miss), or it can flag a safe one (a false alarm). We asked whether a weak checker does better when it is given help. We wrote 48 clinical questions of the kind a GP might ask. Each had one safe answer and two unsafe answers, and each unsafe answer was the safe one with a single error deliberately put in. That made 144 answers. The checker judged every answer in four ways (Figure 3). In direct checking, the baseline, it reads the question and the answer and says safe or unsafe. In fact-by-fact checking, it lists the facts a safe answer must contain before it sees the answer, and then checks the answer against its list. In single-call checking, it writes the list and gives its verdict in one pass, with the answer in view from the start. In checking with critique, a stronger model writes notes on the answer, and the checker reads them before giving its verdict. Each way was run with and without a short summary of the relevant UK guidance placed in front of the checker. Three AI models took the checker's role. The main one was Claude Haiku 5.5, a small and cheap model, and the stronger model that wrote the notes was Claude Opus 5.5. We measured two things each time: the share of unsafe answers that the checker flagged (its catch rate) and the share of safe answers that it wrongly flagged (its false-alarm rate). None of the three kinds of help clearly raised the catch rate without also raising the false-alarm rate. With fact-by-fact checking, the main checker caught 98.4% of unsafe answers, up from 88.5% with direct checking, but it also wrongly flagged 68.1% of safe answers, up from 11.8%. Before the experiment we had decided that fact-by-fact checking would count as an improvement only if false alarms rose by less than 10 percentage points. They rose by far more, so by our own test it failed. A checker that flags most safe answers tells the doctor reading it very little. Most of the extra catches were of one kind of error, where something important had been left out of the answer. The cheapest help was the guidance summary, which lowered false alarms for two of the three checkers without a clear change in catch rate. The stronger model's notes passed its mistakes on to the checker, mostly in one direction. The stronger model also judged every answer itself, and it made 26 false alarms and 56 misses. Of the 26 false alarms, the main checker repeated 11 when it checked directly and 23 after it had read the notes. Of the 56 misses, it repeated 45 when it checked directly and 14 after it had read the notes. All of these results rest on one doctor's judgement. One of us (KY) decided alone which answers were safe and which were unsafe. The other (OE) then graded a sample of 49 answers without seeing KY's decisions and disagreed on 16, each time calling safe an answer that KY had called unsafe. On the 33 answers we agreed on, the rise in false alarms is still there, but the rise in catches is not confirmed.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24378737
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Helping a Weak AI Checker: More Catches, Many More False Alarms

Kehinde Yahaya, Onyerem Ewuzie
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
preprint

Helping a Weak AI Checker: More Catches, Many More False Alarms

Kehinde Yahaya, Onyerem Ewuzie
preprint en

Abstract

Doctors now ask AI tools what to do about a patient, and they rarely have time to check each answer against the guidelines before acting on it. One safeguard is a second AI model, the checker, which reads the first one's answer and says whether it is safe. To save money and time, the checker is often a smaller and less capable model than the one it checks. It can be wrong in two ways: it can pass an unsafe answer (a miss), or it can flag a safe one (a false alarm). We asked whether a weak checker does better when it is given help. We wrote 48 clinical questions of the kind a GP might ask. Each had one safe answer and two unsafe answers, and each unsafe answer was the safe one with a single error deliberately put in. That made 144 answers. The checker judged every answer in four ways (Figure 3). In direct checking, the baseline, it reads the question and the answer and says safe or unsafe. In fact-by-fact checking, it lists the facts a safe answer must contain before it sees the answer, and then checks the answer against its list. In single-call checking, it writes the list and gives its verdict in one pass, with the answer in view from the start. In checking with critique, a stronger model writes notes on the answer, and the checker reads them before giving its verdict. Each way was run with and without a short summary of the relevant UK guidance placed in front of the checker. Three AI models took the checker's role. The main one was Claude Haiku 5.5, a small and cheap model, and the stronger model that wrote the notes was Claude Opus 5.5. We measured two things each time: the share of unsafe answers that the checker flagged (its catch rate) and the share of safe answers that it wrongly flagged (its false-alarm rate). None of the three kinds of help clearly raised the catch rate without also raising the false-alarm rate. With fact-by-fact checking, the main checker caught 98.4% of unsafe answers, up from 88.5% with direct checking, but it also wrongly flagged 68.1% of safe answers, up from 11.8%. Before the experiment we had decided that fact-by-fact checking would count as an improvement only if false alarms rose by less than 10 percentage points. They rose by far more, so by our own test it failed. A checker that flags most safe answers tells the doctor reading it very little. Most of the extra catches were of one kind of error, where something important had been left out of the answer. The cheapest help was the guidance summary, which lowered false alarms for two of the three checkers without a clear change in catch rate. The stronger model's notes passed its mistakes on to the checker, mostly in one direction. The stronger model also judged every answer itself, and it made 26 false alarms and 56 misses. Of the 26 false alarms, the main checker repeated 11 when it checked directly and 23 after it had read the notes. Of the 56 misses, it repeated 45 when it checked directly and 14 after it had read the notes. All of these results rest on one doctor's judgement. One of us (KY) decided alone which answers were safe and which were unsafe. The other (OE) then graded a sample of 49 answers without seeing KY's decisions and disagreed on 16, each time calling safe an answer that KY had called unsafe. On the 33 answers we agreed on, the rise in false alarms is still there, but the rise in catches is not confirmed.

Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon