Consensus Is Not a Label: Red-Line Invariants for Multi-Model Adjudication Pipelines

Multi-model pipelines — systems in which several large language models draft, critique, judge and verify one another's answers — are increasingly used as evaluation and quality-assurance machinery for high-stakes questions. Their outputs are often summarised with a single reassuring word: consensus. This report argues that consensus in such a pipeline is not a property of the final answer but of the invariants enforced at every stage, and that these invariants fail in characteristic, auditable ways. We describe the design of a production multi-model adjudication system ("the Club pipeline"): a pool of models from multiple vendors answers a task, cross-examines competing answers, is packaged blind for a panel of independent judges, and is subjected to adversarial verification before a consensus verdict is computed. We then report an independent audit of that pipeline against its own stated invariants. The audit found four red-line violations, all in code, none exotic: confidential tasks leaked to cloud models at the judging and verification stages although the answering stage was correctly local-only; the judging and verification stages ran entirely outside the budget guard, making their cost invisible; a terminal "gate" path stamped consensus.achieved from the single drafter's self-reported confidence; and judge/verifier independence was enforced by model alias but not by vendor or by composite adapter. We present the invariant set, the findings mapped to it, the fixes with their regression tests, and the limits of a single-system case study. The broader claim is methodological: any multi-model evaluation pipeline should be required to state its red-line invariants in advance and to survive an adversarial audit against them, because the failure modes that corrupt a verdict are usually plumbing, not modelling. Preprint. Not peer-reviewed.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24173966
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Consensus Is Not a Label: Red-Line Invariants for Multi-Model Adjudication Pipelines

Олександр Мінаков
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Consensus Is Not a Label: Red-Line Invariants for Multi-Model Adjudication Pipelines

Олександр Мінаков
preprint en

Abstract

Multi-model pipelines — systems in which several large language models draft, critique, judge and verify one another's answers — are increasingly used as evaluation and quality-assurance machinery for high-stakes questions. Their outputs are often summarised with a single reassuring word: consensus. This report argues that consensus in such a pipeline is not a property of the final answer but of the invariants enforced at every stage, and that these invariants fail in characteristic, auditable ways. We describe the design of a production multi-model adjudication system ("the Club pipeline"): a pool of models from multiple vendors answers a task, cross-examines competing answers, is packaged blind for a panel of independent judges, and is subjected to adversarial verification before a consensus verdict is computed. We then report an independent audit of that pipeline against its own stated invariants. The audit found four red-line violations, all in code, none exotic: confidential tasks leaked to cloud models at the judging and verification stages although the answering stage was correctly local-only; the judging and verification stages ran entirely outside the budget guard, making their cost invisible; a terminal "gate" path stamped consensus.achieved from the single drafter's self-reported confidence; and judge/verifier independence was enforced by model alias but not by vendor or by composite adapter. We present the invariant set, the findings mapped to it, the fixes with their regression tests, and the limits of a single-system case study. The broader claim is methodological: any multi-model evaluation pipeline should be required to state its red-line invariants in advance and to survive an adversarial audit against them, because the failure modes that corrupt a verdict are usually plumbing, not modelling. Preprint. Not peer-reviewed.

Zenodo (CERN European Organization for Nuclear Research)
Peace, justice, and strong institutions
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon