Paper Draft. What an Unreliable Verifier Collects (v10)
A selector that picks among n candidate runs with a verifier can reach the oracle, the share of tasks that at least one candidate solves, only if the verifier never accepts a wrong candidate. This draft replaces the single error rate of the earlier working paper by two rates per task, false accept α and false reject β, derives what follows (a precision ceiling, the asymmetry of α and β, per-task rates), and measures the rates on a released model verifier (R2E-Gym on 500 SWE-bench Verified bugs) and in two pre-registered runs of a Claude Haiku judge and two test writers over the same 1,600 Qwen3-Coder candidates on SWE-rebench. A verifier's two error rates, measured per task, predict what a selector collects; on agent tasks the prize goes to a verifier with a small false-accept rate, and neither model judges nor model-written tests, blind or reading the code, supply one, so the best deployable rule found collects 0.28 of the prize at eight candidates by gating on tests and ranking by a judge. Companion to the note "What an Unreliable Verifier Collects" (https://alatip.github.io/what-an-unreliable-verifier-collects/) and supplement to the working paper "Exploring the Boundaries of Multi-Agent Ensembles" (doi:10.5281/zenodo.23311676). Draft v10: sections 0–4 are the plan before the experiments, 5–6 the hypotheses and plan with verdicts, 7–9 and the Conclusion the results; 30 references.
Authors
- Aziz Latipov (ORCID: https://orcid.org/0009-0003-8938-8223)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-13
- DOI
- https://doi.org/10.5281/zenodo.23336059
- Primary Topic
- Software Engineering Research
- Type
- preprint