Mean-Field Aided QMIX: A Scalable and Flexible Q-Learning Approach for Large-Scale Agent Groups

Value decomposition methods are effective for multi-agent reinforcement learning (MARL), with QMIX being one of the most advanced. However, it struggles with scalability and flexibility in large-scale agent systems. The introduction of mean-field theory into MARL provides a potential solution to both scalability and flexibility challenges. In this paper, we propose Mean-Field Aided QMIX (MF-QMIX), a novel algorithm that addresses these challenges by incorporating mean-field theory into the QMIX framework. MF-QMIX significantly reduces the computational complexity from O(n) in QMIX to O(1), making it highly scalable and suitable for large-scale environments with many agents. Additionally, MF-QMIX is flexible, allowing the trained model to be applied across various group sizes without retraining, thus accommodating dynamic changes in the number of agents. Our extensive experiments demonstrate that MF-QMIX outperforms existing methods, both in computational efficiency and adaptability, across cooperative and competitive scenarios. This establishes MF-QMIX as a scalable and flexible solution for large-scale MARL problems.