The Dark Room in the Reward Channel: How Dense Prediction Rewards Destroy LLM Agents under Group-Normalized RL—and What Actually Works

Dense per-step supervision is an appealing fix for sparse-reward long-horizon LLM agents: reward the agent for predicting its next observation, and memory should follow. We show that under group-normalized RL (GRPO), this recipe does not merely fail—it actively destroys the policy. Across three model scales (1.7B/4B/8B) on ALFWorld, a potential-based prediction reward drives every run into a degenerate absorbing state: prediction accuracy →1.0,task success →0, episode length pinned at the horizon—an empirical instantiation of the "dark room" pathology. We causally localize the mechanism with a single-factor rescue: removing GRPO's std-normalization (the only change) turns the same reward from catastrophic (0%) into baseline parity. In all-fail groups the normalized advantage is invariant to the shaping coefficient (Proposition 1), which jointly explains full-scale amplification and the failure of annealing. We characterize the failure across scales (non-monotonic collapse timing, joint early-warning signature, irreversibility), separate coverage/dynamics/capacity axes in a synthetic POMDP with exactly computable feature coverage, and run a controlled signal-delivery matrix: every reward-channel variant is at best neutral, while the auxiliary-loss channel gains ~20 points—and a shuffled-gold placebo matching it shows exact content–context pairing is not the active ingredient. We distill these results into a variance-profile criterion: group z-scoring amplifies a dense signal's within-group variance at saturation, so signals whose variance vanishes at mastery are structurally amplifier-safe. All endpoints are single-seed; seed replication and further controls are preregistered and in progress.