Sample Efficient Experience Replay in Non-Stationary Environments

Reinforcement learning (RL) in non-stationary environments presents significant challenges as distribution shifts in transition dynamics and rewards rapidly invalidate previously gathered experience. Traditional Experience replay (ER) methods that prioritize solely by temporal difference error (TD-error) conflate policy-induced distribution shift with exogenous environment changes, yielding suboptimal sampling and slow adaptation. To address this limitation, we introduce the Discrepancy of Environment (DoE), a metric that isolates the impact of environment transitions on value estimation. Building on DoE, we further propose the Discrepancy of Environment Prioritized Experience Replay (DEER), which jointly prioritizes samples by policy evolution and detected environmental variation. DEER employs a binary change-point detector to signal shifts and, conditional on detection, switches between pre- and post-shift prioritization tailored to the current dynamics. This design improves sample efficiency and stabilizes learning under dynamic conditions. Across four non-stationary benchmarks, DEER improves off-policy performance by 11.54% on average relative to the strongest ER baseline.