Reward-Guided Subspace Fusion and Spotlight for Multi-Agent State Space Exploration

Efficient exploration towards high-dimensional state spaces of multiple agents is a long-standing challenge in reinforcement learning, since the size of the state space grows exponentially with the number of agents and state dimensions. Recent studies have shown that most state spaces can be partitioned into a set of subspaces with different degrees of importance for the policy improvement. Therefore, this work proposes Subspace Grouping for Exploration (SGE), a novel approach to spotlight the exploration on those subspaces that contribute the most to policy improvements for increasing the exploration efficiency. Specifically, a metric is designed to quantify the importance of each subspace based on the variance of data points within it. Recognizing that multiple candidate subspaces often exhibit similar importance scores, we propose fusing these subspaces to reduce complexity and further improve exploration efficiency. <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$k$</tex>-means clustering is employed to automatically identify and merge candidate subspaces with similar importance into larger, more representative subspaces. Finally, the proposed method is incorporated into the existing reward-based exploration framework, which will give the agents a high reward once they visit a new subspace, thereby encouraging their appearance of exploration behaviors. Empirical results on a series of StarCraft multi-agent challenge (SMAC) and Multi-Agent MuJoCo tasks demonstrate that our method enhances exploration efficiency in high-dimensional environments compared to existing approaches.