Checking the Figures AI Systems Publish About Themselves: A Minimal Audit Protocol and Its First Two Applications

A system that publishes figures about its own quality makes claims a reader can check. This paper sets out a minimal protocol for checking them and reports its first two applications. The protocol, published separately as version 0.2, has thirteen checks, four supporting techniques and a rule that classifies every finding as a measurement, omission or framing problem. The first application was a self-audit of GaiaLab, a deployed biomedical hypothesis-generation system built and operated by the author. It produced seven published corrections. The interface displayed a grounding value of 1.00 on 46.7% of 5,504 runs, while the system's own share-readiness flag was set on 0.56% of them. The second application was to Biomedical-RAG-Agent, an open-source retrieval-augmented question-answering harness the author did not build. With providers pinned, the harness reproduced its published figures within the tolerance set before the run, at an API cost of $7.51. All six findings concerned reporting. The report's headline accuracy lift of 18 points over a closed-book baseline is +0.00 to +0.04 over a classifier that always answers "yes" (SE 0.04, n = 150). On macro-F1, which the report publishes in its body, retrieval improves on every trivial comparator by +0.16 to +0.18. The six core checks transferred from the first system to the second. The second audit added three reproducibility checks. Its target had nothing for the four conditional checks to run against. Two of the first system's corrections fall outside the protocol's coverage and are listed as such. The second system's author was sent the findings before publication and did not respond within the fourteen days the protocol allows.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-21
DOI
https://doi.org/10.5281/zenodo.24414182
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Checking the Figures AI Systems Publish About Themselves: A Minimal Audit Protocol and Its First Two Applications

oluwafemi Idiakhoa
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Checking the Figures AI Systems Publish About Themselves: A Minimal Audit Protocol and Its First Two Applications

oluwafemi Idiakhoa
preprint en

Abstract

A system that publishes figures about its own quality makes claims a reader can check. This paper sets out a minimal protocol for checking them and reports its first two applications. The protocol, published separately as version 0.2, has thirteen checks, four supporting techniques and a rule that classifies every finding as a measurement, omission or framing problem. The first application was a self-audit of GaiaLab, a deployed biomedical hypothesis-generation system built and operated by the author. It produced seven published corrections. The interface displayed a grounding value of 1.00 on 46.7% of 5,504 runs, while the system's own share-readiness flag was set on 0.56% of them. The second application was to Biomedical-RAG-Agent, an open-source retrieval-augmented question-answering harness the author did not build. With providers pinned, the harness reproduced its published figures within the tolerance set before the run, at an API cost of $7.51. All six findings concerned reporting. The report's headline accuracy lift of 18 points over a closed-book baseline is +0.00 to +0.04 over a classifier that always answers "yes" (SE 0.04, n = 150). On macro-F1, which the report publishes in its body, retrieval improves on every trivial comparator by +0.16 to +0.18. The six core checks transferred from the first system to the second. The second audit added three reproducibility checks. Its target had nothing for the four conditional checks to run against. Two of the first system's corrections fall outside the protocol's coverage and are listed as such. The second system's author was sent the findings before publication and did not respond within the fourteen days the protocol allows.

Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
Checking the Figures AI Systems Publish About Themselves: A Minimal Audit Protocol and Its First Two Applications — oluwafemi Idiakhoa · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS