Investigating Primacy Bias in Multi-Agent Reinforcement Learning: An Empirical Study

In the past two years, the phenomenon of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">primacy bias</i> in reinforcement learning has been extensively investigated, sparking discussions within the community regarding the plasticity of reinforcement learning. This study represents the first comprehensive exploration of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">primacy bias</i> in multiagent reinforcement learning. Building on our previous works, we demonstrate that multi-agent reinforcement learning also encounters the challenge of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">primacy bias</i>. We then provide a comparative analysis across various settings, including different evaluation methods, the sharing of policy parameters, and the adaptability of decentralized policies. We conducted extensive experiments on multiple multi-agent benchmarks. Our findings reveal specific characteristics of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">primacy bias</i> in multi-agent learning, showing the difference between them with those in single-agent reinforcement learning. While, we discuss the limitations and challenges encountered when directly applying existing solutions from the reinforcement learning domain to multi-agent scenarios.