Residual Reinforcement Learning Enhanced With Unsuccessful Episode Buffer

Reinforcement learning (RL) agents have proven to be highly effective in many control domains; however, despite their generally promising performance, they may still encounter difficulties in certain challenging scenarios. To address this limitation, we propose a two-stage learning framework, where an initial pre-trained policy is extended with a second set of networks, aiming to correct the base agent's knowledge and preserve its behavior. Both modules are trained with the conventional RL loop, while the second learning system is also faced with an increased number of training samples from those problematic scenarios to provide more exploration possibilities across them. We evaluated the effectiveness of the proposed framework on various environments from the robotic and vehicle control domains. The results showed that the final agent demonstrates improved performance and can successfully complete a broader range of scenarios compared to the baseline agents, while also maintaining the control quality.