Multi-agent cooperative confrontation with proximal policy optimization in urban environments

To address the issue that urban environments often make command systems inefficient and inflexible due to their geospatial complexity and dynamic changes, a multi-agent cooperative confrontation method with proximal policy optimization for urban environments was proposed. First, on the basis of establishing a comprehensive urban confrontation environment, the AC (actor-critic) network with proximal policy optimization was used to solve the problem. Then, aiming at the multi-to-one critic network, an embedding method was adopted to address the issue of evaluating the decision-making of heterogeneous agents with different spatial dimensions. Furthermore, adaptive sampling was added to assist in the updating of proximal policy optimization. Finally, the weights of the actor network were inherited to help agents quickly take over the corresponding tasks. Experimental results show that the proposed method improves 22.67% reward and 8.14% convergence rate compared to other methods, which not only meets the decision-making of multiple agents’ cooperative confrontation in urban environments, but also is compatible with the cooperative confrontation of multiple heterogeneous agents.