Balanced Subgoals Generation in Hierarchical Reinforcement Learning

Hierarchical Reinforcement Learning(HRL) has exhibited potential in scaling reinforcement learning methods. However, it encounters difficulties to expand in tasks with sparse external rewards. Among them, the issues of exploration inefficiency and non-stationarity are serious. In this paper, we propose a novel HRL approach in which we design two measures for subgoals: balance and potentiality. By generating a new subgoal and strategically extending it toward more promising regions, we enhance exploration efficiency by striking a balance in the process. Lastly, leveraging the two measures, we employ an active exploration policy to prevent the introduction of low-level intrinsic rewards, which naturally avoids non-stationary problems. Experimental results demonstrate that our approach surpasses the current state-of-the-art HRL baselines in MuJoCo tasks with sparse rewards.