Agreement among large language models does not track medical truth in health claim verification

Large language models (LLMs) are increasingly proposed as tools to verify online health information. A common assumption from the self-consistency and ensemble literature is that agreement across models signals reliability. But it remains untested whether inter-model consensus tracks medical truth. We evaluated 10 LLMs spanning frontier, Russian-specific and medical-specialized families on RuHealth, a corpus of 108 short health claims (40 true, 40 false, 28 disputed) validated through multi-stage physician review. Each claim was classified as true , false , or disputed , in both Russian and a validated English translation, across decoding temperatures and 3 stochastic runs ( \({\sim }\) 90,000 judgements). Frontier models were most accurate (macro-F1 64–70) but no model was reliable: “disputed” was assigned to 59.8% of judgements versus a 26% true base rate, while false claims were systematically missed. Majority voting did not help. Three frontier models were unanimous on 90 of 108 claims, yet 31% of those unanimous verdicts were wrong, and 26 of 28 assigned “disputed” to claims that were actually true (14) or false (12). Thus, in this setting on health claims, agreement among contemporary LLMs is not evidence of medical correctness; this effect, which we term consensus mirage , proves to be robust to prompt wording and language.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-14
DOI
https://doi.org/10.1038/s41598-026-79438-6
Primary Topic
Misinformation and Its Impacts
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Agreement among large language models does not track medical truth in health claim verification

Nikolenko Sergey, Anna Probst, Anton Surkov, Elena Artemenko et al.
4 links
Scientific Reports
Misinformation and Its Impacts
article

Agreement among large language models does not track medical truth in health claim verification

Nikolenko Sergey, Anna Probst, Anton Surkov, Elena Artemenko, Olessia Koltsova
article en

Abstract

Large language models (LLMs) are increasingly proposed as tools to verify online health information. A common assumption from the self-consistency and ensemble literature is that agreement across models signals reliability. But it remains untested whether inter-model consensus tracks medical truth. We evaluated 10 LLMs spanning frontier, Russian-specific and medical-specialized families on RuHealth, a corpus of 108 short health claims (40 true, 40 false, 28 disputed) validated through multi-stage physician review. Each claim was classified as true , false , or disputed , in both Russian and a validated English translation, across decoding temperatures and 3 stochastic runs ( \({\sim }\) 90,000 judgements). Frontier models were most accurate (macro-F1 64–70) but no model was reliable: “disputed” was assigned to 59.8% of judgements versus a 26% true base rate, while false claims were systematically missed. Majority voting did not help. Three frontier models were unanimous on 90 of 108 claims, yet 31% of those unanimous verdicts were wrong, and 26 of 28 assigned “disputed” to claims that were actually true (14) or false (12). Thus, in this setting on health claims, agreement among contemporary LLMs is not evidence of medical correctness; this effect, which we term consensus mirage , proves to be robust to prompt wording and language.

Scientific Reports
National Research University Higher School of Economics (RU)
National Research University Higher School of Economics
Misinformation and Its Impacts
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Agreement among large language models does not track medical truth in health claim verification — Nikolenko Sergey, Anna Probst, et al. · Scientific Reports (2026) | TGRS Research Map | TGRS