Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective

While Reinforcement Learning with Verifiable Rewards (RLVR) can enhance LLM reasoning, its training process carries a critical risk: entropy collapse. This phenomenon is a rapid decrease in policy entropy, which severely limits exploration and diminishes learning effectiveness. Recent methods attempt to mitigate this collapse via heuristic entropy interventions, yet the underlying mechanisms governing entropy remain unclear. In this work, we conduct a theoretical and quantitative analysis of GRPO's entropy dynamics, revealing that token-level entropy change in each update step is jointly governed by four key factors: clipping strategy, advantage, token probability, and token entropy. These findings not only explain the mechanisms of existing methods, but also reveal their limitations: they rely on heuristic adjustments to only one or two factors, leaving other relevant factors unconsidered and reducing their effectiveness. This motivates us to propose a new method, STEER, which adaptively reweights tokens based on their estimated entropy change to regulate entropy in a principled manner. Experiments on both math and coding benchmarks demonstrate that STEER effectively mitigates entropy collapse and consistently outperforms state-of-the-art baselines.