Centralized Multi-Agent Proximal Policy Optimization with Attention

This paper introduces a novel centralized multi-agent framework employing proximal policy optimization (PPO), a state-of-the-art reinforcement learning algorithm. Multiple subagents focus on different aspects of the environment, and the actions suggested by the subagents are used to augment the environment space for a superagent that encompasses the subagents' different approaches. A key feature of this model is the built-in attention module that balances the weight attributed to the environment variables and the suggested actions. This architecture is designed to promote emergent behaviour and exploration, completing the standard training of a PPO agent. We study the efficiency and decision accuracy from the model across several MuJoCo scenarios, and perform an ablation study to demonstrate the influence of the attention mechanism. Our results indicate a tradeoff in performance linked to the dimensions of the action space and address the ideal use case for this framework.