Remember Your Best: Improving Exploitation in Deep Q-Network

Reinforcement Learning (RL) deals with developing algorithms for an autonomous agent to learn decisions that maximize a long-term goal in sequential decision-making problems. A challenging aspect of RL is the sparse reward setting, where the agent receives positive feedback only when the goal is achieved, with sparse intermediate feedback in between. In these environments, the agent must explore various actions to learn how to achieve the goal. Moreover, it is equally essential for the agent to reinforce its previous best experiences for faster convergence to the optimal solution. This process is referred to as exploitation. We argue that, in RL algorithms that use bootstrapping, the traditional way of exploitation does not necessarily facilitate optimal behavior. This work proposes a simple exploitation strategy that “remembers” previous best experiences. Specifically, we train, in parallel, a neural network to predict actions that led to the best outcomes thus far. Subsequently, the actions from this neural network are integrated during the exploitation stages of learning. We demonstrate that our proposed algorithm performs better than its vanilla counterpart through empirical studies on three standard RL baselines.