Pre-Registered, Budget-Certified Verdicts for the Test and Evaluation of AI Systems: Certified Union Constants, Attempt Budgets, Commit-Reveal Labels and Certified Operating Envelopes

A test-and-evaluation verdict on an AI system is usually a point estimate compared with a threshold, and it can be manufactured: by re-running the evaluation until it passes, by re-reading a test after every batch, by re-phrasing the question after seeing the data, or by letting the system see its labels. We describe a protocol in which a verdict is a certificate that cannot be manufactured in these ways. Each claim is decided with a concentration margin whose logarithmic constant must cover a declared budget of looks and attempts; the evaluator certifies that constant in exact integer arithmetic before any data and refuses an under-declared policy. A canonical digest of what is asked is registered before the data; labels are committed by the source before the system commits its outputs; verdicts are three-valued (Admitted, Refuted, Open); and the same final looks certify the operating envelope, the limiting condition and the break point at no extra statistical cost. We prove a per-claim guarantee across all budgeted attempts, give a sign-pattern calibration bound 2.87 times tighter than the binned bound in a representative configuration, and report 72 verdicts in eight evaluation modes with none contrary to the declared truth. All quantities are bit-exact across architectures. Preprint, version 1, not peer reviewed. AI assistance is disclosed in the paper.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-20
DOI
https://doi.org/10.5281/zenodo.24384615
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Pre-Registered, Budget-Certified Verdicts for the Test and Evaluation of AI Systems: Certified Union Constants, Attempt Budgets, Commit-Reveal Labels and Certified Operating Envelopes

Anthony Moreno
4 links
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Pre-Registered, Budget-Certified Verdicts for the Test and Evaluation of AI Systems: Certified Union Constants, Attempt Budgets, Commit-Reveal Labels and Certified Operating Envelopes

Anthony Moreno
preprint en

Abstract

A test-and-evaluation verdict on an AI system is usually a point estimate compared with a threshold, and it can be manufactured: by re-running the evaluation until it passes, by re-reading a test after every batch, by re-phrasing the question after seeing the data, or by letting the system see its labels. We describe a protocol in which a verdict is a certificate that cannot be manufactured in these ways. Each claim is decided with a concentration margin whose logarithmic constant must cover a declared budget of looks and attempts; the evaluator certifies that constant in exact integer arithmetic before any data and refuses an under-declared policy. A canonical digest of what is asked is registered before the data; labels are committed by the source before the system commits its outputs; verdicts are three-valued (Admitted, Refuted, Open); and the same final looks certify the operating envelope, the limiting condition and the break point at no extra statistical cost. We prove a per-claim guarantee across all budgeted attempts, give a sign-pattern calibration bound 2.87 times tighter than the binned bound in a representative configuration, and report 72 verdicts in eight evaluation modes with none contrary to the declared truth. All quantities are bit-exact across architectures. Preprint, version 1, not peer reviewed. AI assistance is disclosed in the paper.

Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon