Reward Hacking on TRIAL: Tracing Evidence Across Agent Actions

Reward hacking poses severe risks across reinforcement learning, automated workflows, and benchmark evaluation, where agents exploit unintended shortcuts to maximize rewards without demonstrating genuine capabilities. Detecting these shortcuts is hard: across complex, long-horizon tasks, actions unfold over multi-hop transformations, while lengthy traces conceal dependencies from language model judges. To provide verifiable integrity guarantees, we introduce TRIAL, a trace checker that audits agent logs through provenance tracing. TRIAL compiles recorded actions into a causal dependency graph of versioned resources and applies deterministic policy rules to verify whether protected content reached the scored artifact. Across ten public corpora containing 94,162 runs, TRIAL reliably recovers documented exploits: on Terminal-Bench 2.1, all 17 flagged entry–task pairs match upheld maintainer decisions; on Terminal-Bench 2.0, withholding credit for dependency findings drops a compromised top-8 entry from rank 8 to 30 (rank 75 with all flags, reducing success from 83.1% to 4.3%); and on PostTrainBench, TRIAL outperforms a high-reasoning GPT-5.5 judge by 29.7 F1 points. These results demonstrate that auditing reward hacking requires tracking multi-hop provenance across versioned resources while exposing the evidence behind each finding.Preprint, version v1 (manuscript dated 2026-10-14). Landing page with full-text HTML: https://jeremyjc67.github.io/papers/trial/