Where AI mathematical discoveries fail: approximate verifiers and invalid results

Automated search systems now report bounds and constructions for open mathematical problems, and those results are accepted by an executable verifier. We ask where such results fail. We tested 168 open-source verifiers from AI-for-mathematics systems. Thirty-four (20.2%, 95% CI 14.4-27.1%) accepted at least one object that is mathematically invalid. On a public leaderboard (snapshot of 26 October 2026), every invalid top-10 entry we found sits on a verifier that samples, stops at a fixed range, clamps, or uses a float tolerance. Three of 29 problems are affected, accounting for 13 of 231 entries. We certify the scores of 28 board entries (24 exactly, 4 by rigorous interval) and give an exactly certified construction for the one problem whose top-10 board entries all fail. We also report negative results: a generic fuzzer predicts exploitability poorly, and in an interim pre-registered comparison a 14B local model did not gain score from the clamp in any of four seeds. In a public library of reinforcement-learning environments, a pre-registered test of 14 reward functions found, after we corrected for defects in our own oracle, one that gives full credit to a constant wrong answer.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-27
DOI
https://doi.org/10.5281/zenodo.23681091
Primary Topic
Software Testing and Debugging Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Where AI mathematical discoveries fail: approximate verifiers and invalid results

Advatiya Thalluri
3 links
Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
preprint

Where AI mathematical discoveries fail: approximate verifiers and invalid results

Advatiya Thalluri
preprint en

Abstract

Automated search systems now report bounds and constructions for open mathematical problems, and those results are accepted by an executable verifier. We ask where such results fail. We tested 168 open-source verifiers from AI-for-mathematics systems. Thirty-four (20.2%, 95% CI 14.4-27.1%) accepted at least one object that is mathematically invalid. On a public leaderboard (snapshot of 26 October 2026), every invalid top-10 entry we found sits on a verifier that samples, stops at a fixed range, clamps, or uses a float tolerance. Three of 29 problems are affected, accounting for 13 of 231 entries. We certify the scores of 28 board entries (24 exactly, 4 by rigorous interval) and give an exactly certified construction for the one problem whose top-10 board entries all fail. We also report negative results: a generic fuzzer predicts exploitability poorly, and in an interim pre-registered comparison a 14B local model did not gain score from the clamp in any of four seeds. In a public library of reinforcement-learning environments, a pre-registered test of 14 reward functions found, after we corrected for defects in our own oracle, one that gives full credit to a constant wrong answer.

Zenodo (CERN European Organization for Nuclear Research)
Software Testing and Debugging Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon