Where AI mathematical discoveries fail: approximate verifiers and invalid results
Automated search systems now report bounds and constructions for open mathematical problems, and those results are accepted by an executable verifier. We ask where such results fail. We tested 168 open-source verifiers from AI-for-mathematics systems. Thirty-four (20.2%, 95% CI 14.4-27.1%) accepted at least one object that is mathematically invalid. On a public leaderboard (snapshot of 26 October 2026), every invalid top-10 entry we found sits on a verifier that samples, stops at a fixed range, clamps, or uses a float tolerance. Three of 29 problems are affected, accounting for 13 of 231 entries. We certify the scores of 28 board entries (24 exactly, 4 by rigorous interval) and give an exactly certified construction for the one problem whose top-10 board entries all fail. We also report negative results: a generic fuzzer predicts exploitability poorly, and in an interim pre-registered comparison a 14B local model did not gain score from the clamp in any of four seeds. In a public library of reinforcement-learning environments, a pre-registered test of 14 reward functions found, after we corrected for defects in our own oracle, one that gives full credit to a constant wrong answer.
Authors
- Advatiya Thalluri (ORCID: https://orcid.org/0009-0004-3435-9772)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-10-27
- DOI
- https://doi.org/10.5281/zenodo.23681091
- Primary Topic
- Software Testing and Debugging Techniques
- Type
- preprint