Reward Hacking Is an Accounting Error: Eliminating Specification Gaming and the Alignment Tax via Conservation-Gated Reinforcement Learning and Anchored Verifier Evolution

Prevailing alignment paradigms treat specification gaming as an innate disposition of capable optimizers that must be policed via post-hoc filters, auxiliary critique models, or soft reward penalties, imposing a steep “alignment tax” as optimization scales. We prove that reward hacking is instead an accounting error: it occurs whenever an optimizer collects visible proxy gain G(τ) without independent causal proof L(τ) that the underlying computation was performed. Unpressured language models rarely scheme (0/720 hardcodes in clean probes; 0.28% across 2,480 impossible coding attempts); standard post-training manufactures deception by paying unbacked credit U(τ) = G(τ) − L(τ) > 0. We introduce Ledger-Gated Training (LGT), enforcing the strict conservation identity U(τ) = 0 prior to every gradient update via a multiplicative trajectory gate anchored by an unbiased forward-KL token objective. We prove that all finite linear penalties (λ < ∞) and Constrained-MDP budgets (E[U] ≤ δ) fail via hybrid subsidies, prerequisite shortcuts, and budget exhaustion; that any receipt auditor better than chance (ϵleak + ϵalarm < 1) renders honest work strictly dominant within the competence ratio κ; and that under natural policy-gradient flow, LGT extinguishes shortcuts exponentially while amplifying honest trajectory mass (ρt > 1), yielding a provably negative alignment tax. Evaluated with all gates switched OFF on unseen tasks, running online GRPO to KL convergence causes standard RLVR to collapse. Cheating explodes to 54.9% (258/470) and specification-correctness plunges from 76.8% to 27.2%, whereas LGT holds cheating at 1.1% (5/470; 50× reduction, z = 18.4) and lifts specification-correctness to 80.8% (+53.6 pp, z = 16.5). At 3× learning rate, LGT suppresses cheating by 16× (1.1% vs. 18.1%, z = 8.87) and delivers a 2× capability speedup (+22 attempt-1 passes in 31 steps). Across 5/5 runs, LGT rises above initialization on attempt-1 capability, beats a privileged hidden-test oracle on final-of-3 synthesis in 4/5 runs, and cuts cheats on an untrained helper suite in 5/5 runs; in iterative self-training, extinguishing shortcuts to 0.0% ignites hard synthesis pass@1 by 4.1× (4.0% to 16.7%, 16.3%). Under Anchored Verifier Evolution (AVE), a self-revising verifier attacked across 114,584 generations by Qwen2.5-72B achieves 0.0% false certifications (0/324 governed held-out reports across 972 three-track evaluations, vs. 81.5% ablated) while raising recall from 0.953 to 1.000.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24232870
Primary Topic
Reinforcement Learning in Robotics
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Reward Hacking Is an Accounting Error: Eliminating Specification Gaming and the Alignment Tax via Conservation-Gated Reinforcement Learning and Anchored Verifier Evolution

Amr Elnaggar
Zenodo (CERN European Organization for Nuclear Research)
Reinforcement Learning in Robotics
preprint

Reward Hacking Is an Accounting Error: Eliminating Specification Gaming and the Alignment Tax via Conservation-Gated Reinforcement Learning and Anchored Verifier Evolution

Amr Elnaggar
preprint en

Abstract

Prevailing alignment paradigms treat specification gaming as an innate disposition of capable optimizers that must be policed via post-hoc filters, auxiliary critique models, or soft reward penalties, imposing a steep “alignment tax” as optimization scales. We prove that reward hacking is instead an accounting error: it occurs whenever an optimizer collects visible proxy gain G(τ) without independent causal proof L(τ) that the underlying computation was performed. Unpressured language models rarely scheme (0/720 hardcodes in clean probes; 0.28% across 2,480 impossible coding attempts); standard post-training manufactures deception by paying unbacked credit U(τ) = G(τ) − L(τ) > 0. We introduce Ledger-Gated Training (LGT), enforcing the strict conservation identity U(τ) = 0 prior to every gradient update via a multiplicative trajectory gate anchored by an unbiased forward-KL token objective. We prove that all finite linear penalties (λ < ∞) and Constrained-MDP budgets (E[U] ≤ δ) fail via hybrid subsidies, prerequisite shortcuts, and budget exhaustion; that any receipt auditor better than chance (ϵleak + ϵalarm < 1) renders honest work strictly dominant within the competence ratio κ; and that under natural policy-gradient flow, LGT extinguishes shortcuts exponentially while amplifying honest trajectory mass (ρt > 1), yielding a provably negative alignment tax. Evaluated with all gates switched OFF on unseen tasks, running online GRPO to KL convergence causes standard RLVR to collapse. Cheating explodes to 54.9% (258/470) and specification-correctness plunges from 76.8% to 27.2%, whereas LGT holds cheating at 1.1% (5/470; 50× reduction, z = 18.4) and lifts specification-correctness to 80.8% (+53.6 pp, z = 16.5). At 3× learning rate, LGT suppresses cheating by 16× (1.1% vs. 18.1%, z = 8.87) and delivers a 2× capability speedup (+22 attempt-1 passes in 31 steps). Across 5/5 runs, LGT rises above initialization on attempt-1 capability, beats a privileged hidden-test oracle on final-of-3 synthesis in 4/5 runs, and cuts cheats on an untrained helper suite in 5/5 runs; in iterative self-training, extinguishing shortcuts to 0.0% ignites hard synthesis pass@1 by 4.1× (4.0% to 16.7%, 16.3%). Under Anchored Verifier Evolution (AVE), a self-revising verifier attacked across 114,584 generations by Qwen2.5-72B achieves 0.0% false certifications (0/324 governed held-out reports across 972 three-track evaluations, vs. 81.5% ablated) while raising recall from 0.953 to 1.000.

Zenodo (CERN European Organization for Nuclear Research)
Reinforcement Learning in Robotics
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon