Exploring the Detection and Mitigation of Reward Hacking from DeepSeek-R1

DeepSeek AI research team made breakthrough progress in verifiable reasoning tasks such as mathematics and code by using rule verification rewards in coordination with group relative policy optimization (GRPO), but reliable reward models are difficult to build on tasks such as writing, and the risk of Reward Hacking still exists outside the verifiable field. Taking DeepSeek-R1 as an anchor, this paper summarizes the progress of detection and mitigation of Reward Hacking from four aspects: formal definition, data set and evaluation indicators, method level, limitations and response. This paper first combs the formal definition of Reward Hacking and the classification framework under the assumption of proxy compression; Then, typical data sets and evaluation indicators are introduced, and the level of task performance and detection mitigation achieved by current mainstream methods is summarized; Finally, the structure boundary of rule reward, the algorithm layer performance of GRPO deviation, and the direction of detection and mitigation are analyzed. The overview shows that Reward Hacking originates from information compression in agent reward optimization, which can verify that accurate rules in the domain can compress Reward Hacking space, while outside the domain, it needs to rely on a collaborative system consisting of KL monitoring, representation detection, algorithm correction and policy constraints.