Paper Draft. What an Unreliable Verifier Collects (v10)

A selector that picks among n candidate runs with a verifier can reach the oracle, the share of tasks that at least one candidate solves, only if the verifier never accepts a wrong candidate. This draft replaces the single error rate of the earlier working paper by two rates per task, false accept α and false reject β, derives what follows (a precision ceiling, the asymmetry of α and β, per-task rates), and measures the rates on a released model verifier (R2E-Gym on 500 SWE-bench Verified bugs) and in two pre-registered runs of a Claude Haiku judge and two test writers over the same 1,600 Qwen3-Coder candidates on SWE-rebench. A verifier's two error rates, measured per task, predict what a selector collects; on agent tasks the prize goes to a verifier with a small false-accept rate, and neither model judges nor model-written tests, blind or reading the code, supply one, so the best deployable rule found collects 0.28 of the prize at eight candidates by gating on tests and ranking by a judge. Companion to the note "What an Unreliable Verifier Collects" (https://alatip.github.io/what-an-unreliable-verifier-collects/) and supplement to the working paper "Exploring the Boundaries of Multi-Agent Ensembles" (doi:10.5281/zenodo.23311676). Draft v10: sections 0–4 are the plan before the experiments, 5–6 the hypotheses and plan with verdicts, 7–9 and the Conclusion the results; 30 references.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-13
DOI
https://doi.org/10.5281/zenodo.23336059
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Paper Draft. What an Unreliable Verifier Collects (v10)

Aziz Latipov
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Paper Draft. What an Unreliable Verifier Collects (v10)

Aziz Latipov
preprint en

Abstract

A selector that picks among n candidate runs with a verifier can reach the oracle, the share of tasks that at least one candidate solves, only if the verifier never accepts a wrong candidate. This draft replaces the single error rate of the earlier working paper by two rates per task, false accept α and false reject β, derives what follows (a precision ceiling, the asymmetry of α and β, per-task rates), and measures the rates on a released model verifier (R2E-Gym on 500 SWE-bench Verified bugs) and in two pre-registered runs of a Claude Haiku judge and two test writers over the same 1,600 Qwen3-Coder candidates on SWE-rebench. A verifier's two error rates, measured per task, predict what a selector collects; on agent tasks the prize goes to a verifier with a small false-accept rate, and neither model judges nor model-written tests, blind or reading the code, supply one, so the best deployable rule found collects 0.28 of the prize at eight candidates by gating on tests and ranking by a judge. Companion to the note "What an Unreliable Verifier Collects" (https://alatip.github.io/what-an-unreliable-verifier-collects/) and supplement to the working paper "Exploring the Boundaries of Multi-Agent Ensembles" (doi:10.5281/zenodo.23311676). Draft v10: sections 0–4 are the plan before the experiments, 5–6 the hypotheses and plan with verdicts, 7–9 and the Conclusion the results; 30 references.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.