Action Selection in Reinforcement Learning with Upgoing Policy Update

This paper proposes the boosted generalized advantage estimation (BGAE), which extends the generalized advantage estimation (GAE) based on proximal policy optimization (PPO) algorithms with a general advantage estimator used in the Atari 2600 game environment. We utilize the preferable feature of the upgoing policy update (UPGO), which is a crucial algorithm behind the well-known multiagent reinforcement learning model, AlphaStar. The proposed method uses UPGO as an action selector to select good actions and amplify their importance. In addition, we discuss the overestimation of the UPGO algorithm in a standard environment. Unlike other approaches that increase model complexity through overestimation, our novel method effectively addresses the overestimation problem in the UPGO method, offering a streamlined solution in a standard learning environment. Implementing UPGO algorithms can improve the convergence speed of conventional PPO learning in the commonly used OpenAI-Gym Atari environment.