Tactical Reward Shaping for Large-Scale Combat by Multi-Agent Reinforcement Learning

Future unmanned battles desperately require intelligent combat policies, and multi-agent reinforcement learning offers a promising solution. However, due to the complexity of combat operations and large size of the combat group, this task suffers from credit assignment problem more than other reinforcement learning tasks. This study uses reward shaping to relieve the credit assignment problem and improve policy training for the new generation of large-scale unmanned combat operations. We first prove that multiple reward shaping functions would not change the Nash Equilibrium in stochastic games, providing theoretical support for their use. According to the characteristics of combat operations, we propose tactical reward shaping (TRS) that comprises maneuver shaping advice and threat assessment-based attack shaping advice. Then, we investigate the effects of different types and combinations of shaping advice on combat policies through experiments. The results show that TRS improves both the efficiency and attack accuracy of combat policies, with the combination of maneuver reward shaping advice and ally-focused attack shaping advice achieving the best performance compared with that of the baseline strategy.