Regret Optimization Experience Replay in Off-Policy Reinforcement Learning

Experience Replay (ER) allows Deep Reinforcement Learning (RL) agent to reuse past experience, as though recall the same Experience repeatedly. ER enables RL algorithm to be trained by reusing previous states, so that RL agent can obtain more accurate value estimations and action selections. Current policy algorithms either have a rule-based replay policy or uniformly replay past experiences, which may be suboptimal. The agent is updated based on the replay data to maximize the cumulative reward, and the replay policy is updated to provide the more valuable experience for the agent. In this work, we propose a novel experience replay algorithm Regret Minimization Experience Replay (RMER), to improve the immediate reward via sampling and ensure certain exploration capability of agent. Finally, we prove the ascendency of RMER with different off- policy algorithms on the suite of Open AI gym continuous control tasks.