A Generative Adversarial Network for Goal Oriented Subgoal Training Exploration and Rewards

Abstract Exploration and long-horizon credit assignment remain central challenges in continuous-control reinforcement learning, particularly as environment complexity increases. We propose a Generative Adversarial Network for Goal-oriented Subgoal Training, Exploration, and Rewards (GANGSTER), a generative subgoal framework that augments off-policy reinforcement learning with a conditional generative adversarial network trained using structured loss components capturing subgoal desirability, reachability, and diversity. GANGSTER is integrated with both Twin Delayed Deep Deterministic Policy Gradient (TD3) and Soft Actor Critic (SAC) without modifying their core update rules. Across multiple Multi-Joint dynamics with Contact (MuJoCo) environments, the approach can improve sample efficiency, post-solution stability, and final policy performance relative to baseline algorithms, with the largest gains observed in high-complexity tasks. Empirical analysis shows that performance improvements scale with environment complexity and that individual loss components play interpretable and complementary roles. In Ant-v5, GANGSTER reduces time-to-solve by approximately ~31\% while increasing final performance by 53\% compared to baseline TD3, and for SAC in the HalfCheetah-v5 environment, GANGSTER increases final performance by 10\% while improving solution consistency to over 99\%.