Policy-Tested and Adaptive Benchmark of Proof Checkers for Reward Hacking in Lean Theorem Proving

We study the proof checkers that set the reward when a language model is trained to prove theorems in Lean 4. Asked to prove a false statement, Claude Haiku says that it is false and proves a corrected statement under the same name. A reward that reads only the code of the whole file pays this swap in 70 to 77% of single samples on our false core statements and in 98% of four-round feedback episodes, and every statement check that we ran rejects every swapped file. In reinforcement learning for formal proofs, the policy is paid by a program's verdict, and not by the truth of the theorem. Published exploits each came with a fix for the exploit that was seen, and the fixes have not been run on each other's exploits, or on attacks that are written to evade them. We consider that the policy is a reward maximiser that writes the whole candidate file, and propose a released suite of static and adaptive attacks, with honest controls, to measure eleven checker designs, including the released checker of LongCat-Flash-Prover and the Lean comparator. We investigate the soundness of each design, its cost in honest proofs that it rejects, and what a policy finds when it is in the loop. Via experimental results, we show that attacks written by hand beat both of the strongest cheap checker stacks, while the comparator rejects all 8 that we could run it on (core suite) and a reference guard rejects all 14, the guard at a cost of 4 of 24 honest Mathlib proofs, and that Claude Haiku found none of these attacks in 368 samples and 140 S-L9 feedback episodes. Apart from one labelled axiom scaffold that only a plain compile paid, the only exploit that a reward paid was the statement swap, on statements that we made false, with one policy and no training. In a registered prompt control, with an instruction to keep the statement, the policy still offers a corrected file that the same reward pays in 51% of core answers, against 77% without it. An instruction to answer a false statement with one decline line brings the paid swap to 0 of 152, at the cost of declining 9 of 32 hard true samples, and on 6 easy true statements the policy swaps none. Thus, for this model and wording, an instruction to keep the statement did not stop the paid swap, and a statement check did.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-18
DOI
https://doi.org/10.5281/zenodo.24346906
Primary Topic
Logic, programming, and type systems
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Policy-Tested and Adaptive Benchmark of Proof Checkers for Reward Hacking in Lean Theorem Proving

Ryan Steubs
1 links
Zenodo (CERN European Organization for Nuclear Research)
Logic, programming, and type systems
preprint

Policy-Tested and Adaptive Benchmark of Proof Checkers for Reward Hacking in Lean Theorem Proving

Ryan Steubs
preprint en

Abstract

We study the proof checkers that set the reward when a language model is trained to prove theorems in Lean 4. Asked to prove a false statement, Claude Haiku says that it is false and proves a corrected statement under the same name. A reward that reads only the code of the whole file pays this swap in 70 to 77% of single samples on our false core statements and in 98% of four-round feedback episodes, and every statement check that we ran rejects every swapped file. In reinforcement learning for formal proofs, the policy is paid by a program's verdict, and not by the truth of the theorem. Published exploits each came with a fix for the exploit that was seen, and the fixes have not been run on each other's exploits, or on attacks that are written to evade them. We consider that the policy is a reward maximiser that writes the whole candidate file, and propose a released suite of static and adaptive attacks, with honest controls, to measure eleven checker designs, including the released checker of LongCat-Flash-Prover and the Lean comparator. We investigate the soundness of each design, its cost in honest proofs that it rejects, and what a policy finds when it is in the loop. Via experimental results, we show that attacks written by hand beat both of the strongest cheap checker stacks, while the comparator rejects all 8 that we could run it on (core suite) and a reference guard rejects all 14, the guard at a cost of 4 of 24 honest Mathlib proofs, and that Claude Haiku found none of these attacks in 368 samples and 140 S-L9 feedback episodes. Apart from one labelled axiom scaffold that only a plain compile paid, the only exploit that a reward paid was the statement swap, on statements that we made false, with one policy and no training. In a registered prompt control, with an instruction to keep the statement, the policy still offers a corrected file that the same reward pays in 51% of core answers, against 77% without it. An instruction to answer a false statement with one decline line brings the paid swap to 0 of 152, at the cost of declining 9 of 32 hard true samples, and on 6 easy true statements the policy swaps none. Thus, for this model and wording, an instruction to keep the statement did not stop the paid swap, and a statement check did.

Zenodo (CERN European Organization for Nuclear Research)
California State University, Dominguez Hills (US)
Logic, programming, and type systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon