Verification Debt in AI-Generated Mathematics: An Empirical Archival and Adversarial Pilot
The prospect of AI producing research claims faster than researchers can validate themrequires a measurement programme rather than extrapolation from manuscript counts. Weundertake a reproducible pilot centred on the October 2026 openai/math mathematicalmanuscript archive and two independent verification activities. We extract a public revisionchronology, distinguish dependent withdrawals from independent failures, independentlyrecheck a source-level citation misidentification and an extraction-induced false alarm, andrun two implementations of a documented signed-count correction. A repository updatecontaining three linked withdrawals, fourteen revised manuscripts, and thirteen furthercitation/version changes was committed 31 hours 5 minutes after the initial public commit;this measures archive latency, not the labour required to discover the defects. The threewithdrawals share one root sign error and thus cannot be treated as independent Bernoullifailures. In a purpose-built control suite, seventeen targeted corruptions were detected andfive out-of-scope semantic strengthenings were not; these engineered counts are a scope test,not an empirical estimate of natural error-detection sensitivity. We provide dated sourceidentifiers, a machine-readable event ledger, a source manifest with Git blob identifiers, and anoffline reproducer. Neither independent reviewer labour times nor downstream scientific-usetimes can currently be estimated from the public material inspected. We give a preregistrableprospective protocol capable of measuring both. The pilot demonstrates the feasibility ofverification observability while refusing population-level error-rate claims unsupported by thedata.
Authors
- Tony Newton (ORCID: https://orcid.org/0009-0009-9189-6431)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-15
- DOI
- https://doi.org/10.5281/zenodo.24247093
- Primary Topic
- Academic Publishing and Open Access
- Type
- preprint