Checking the Figures AI Systems Publish About Themselves: A Minimal Audit Protocol and Its First Two Applications
A system that publishes figures about its own quality makes claims a reader can check. This paper sets out a minimal protocol for checking them and reports its first two applications. The protocol, published separately as version 0.2, has thirteen checks, four supporting techniques and a rule that classifies every finding as a measurement, omission or framing problem. The first application was a self-audit of GaiaLab, a deployed biomedical hypothesis-generation system built and operated by the author. It produced seven published corrections. The interface displayed a grounding value of 1.00 on 46.7% of 5,504 runs, while the system's own share-readiness flag was set on 0.56% of them. The second application was to Biomedical-RAG-Agent, an open-source retrieval-augmented question-answering harness the author did not build. With providers pinned, the harness reproduced its published figures within the tolerance set before the run, at an API cost of $7.51. All six findings concerned reporting. The report's headline accuracy lift of 18 points over a closed-book baseline is +0.00 to +0.04 over a classifier that always answers "yes" (SE 0.04, n = 150). On macro-F1, which the report publishes in its body, retrieval improves on every trivial comparator by +0.16 to +0.18. The six core checks transferred from the first system to the second. The second audit added three reproducibility checks. Its target had nothing for the four conditional checks to run against. Two of the first system's corrections fall outside the protocol's coverage and are listed as such. The second system's author was sent the findings before publication and did not respond within the fourteen days the protocol allows.
Authors
- oluwafemi Idiakhoa (ORCID: https://orcid.org/0009-0008-7911-1171)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-21
- DOI
- https://doi.org/10.5281/zenodo.24414182
- Primary Topic
- Ethics and Social Impacts of AI
- Type
- preprint