Phased Optimal Experience Replay Method

In the field of reinforcement learning, experience replay buffers are frequently used. In previous work, most of the experience replay methods used uniform sampling or preferential sampling based on value of TD-errors. However, while these methods can improve data utilization and accelerate training speed, they still fall short in terms of rapid convergence. In this paper we develop a framework for best experience, so as to replay important transitions more frequently, and learn more efficiently. We implemented optimal experience replay in various state-of-the-art reinforcement learning algorithms, including both discrete and continuous action spaces. Experimental results show that using optimal experience replay can improve the performance of these algorithms.