Trust-Gated Behavioral Auditing: Detecting Reward Hacking in Reinforcement Learning Agents

Reward going up does not mean an agent learned anything. METR (2025) documented frontier models deleting evaluation harnesses, modifying test cases, and editing reward computation code — proxy reward rising throughout. Skalse et al. (NeurIPS 2022) proved this gap is structural: any non-constant proxy reward is hackable. Gao et al. (ICML 2023) quantified the dynamics — proxy and true reward decouple at a phase transition invisible from the reward curve alone. No existing tool performs post-hoc composite behavioral auditing across standard RL environments without per-environment engineering. We introduce the Learning Quality Score (LQS). LQS decomposes agent behavior into independently measurable probes combined via a trust-gated formula. The primary contribution is the trust gate: when systematic exploitation is detected, all behavioral measurements are multiplicatively invalidated. Not penalized — epistemically voided. In a number-sorting environment with a deliberately exploitable reward function, reward ranked a random agent (0.750) above a hacking agent (0.654) and flagged neither; LQS assigned the hacker 0.020, recovering the correct ordering. An LQS-inspired GRPO training signal drove Hack Index 1.00→0.00 in 500 steps while reward increased 46.5% — reward sees a quantitative gain; LQS reveals a qualitative shift. LQS is validated on environments with exploitable reward structures, where behavioral signal is most informative. We introduce the Learning Capacity conjecture: any proxy reward imposes an information-theoretic bound on how much of the true objective an optimizing agent can recover — and LQS estimates where an agent currently sits on the proxy-true reward divergence curve.