Closely Cooperative Multi-Agent Reinforcement Learning Based on Intention Sharing and Credit Assignment

Collaborative tasks are important in multi-agent systems. Multi-agent reinforcement learning is a commonly used technique for solving multi-agent cooperative policy learning. The closely collaborative task is a special but common case within cooperative tasks, where the change in the environmental state requires multiple agents to simultaneously perform specific actions. For example, in a box-pushing task where the boxes are heavy and require multiple agents to push simultaneously. The closely cooperative task faces some unique challenges. Firstly, the completion of a closely collaborative task requires agents to synchronize their actions, necessitating a consistent intention among them. Secondly, when some agents' erroneous actions lead to task failure, it becomes a challenge to avoid incorrectly penalizing agents who performed the correct actions. These challenges make most of the existing MARL methods perform poorly on this task. In this letter, we propose a closely collaborative multi-agent reinforcement learning(CC-MARL) algorithm based on intention sharing and credit assignment. We use a two-phase training to learn intention encoding and intention sharing respectively, and decompose joint action values based on counterfactual baseline ideas. We deployed scenarios in both simulated and real environments with various sizes, numbers of boxes, and numbers of agents and compare CC-MARL with various classical MARL algorithms on box-pushing tasks of different map scales in simulation, demonstrating the state-of-the-art of our method.