Heterogeneous multi-agent collaborative radar countermeasure strategy generation method based on hierarchical reinforcement learning

In radar countermeasure, to enhance the collaborative capability of heterogeneous multi-agent, a hierarchical reinforcement learning-based method is proposed for generating collaborative radar countermeasure strategy. First, a multi-agent radar countermeasure scenario (MRCS) is constructed, and an objective function is formulated from the evaluation of task accomplishment. Then, using a hierarchical reinforcement learning model, a strategy model based on deep neural networks is constructed, comprising a top-level control model and individual execution models. The top-level control model aggregates the detection results and individual information of each agent to form control information, while individual execution models compute discrete action values based on individual detection results and information, selecting the action with the maximum discrete action value as the executed action. The collective value is then formed by aggregating the maximum action values of all agents. Through temporal difference iteration, the deep neural network parameters of the strategy model are optimized to improve the strategy. Then, a strategy optimization method is presented. The experiments are divided into two parts, conducted in StarCraft Multi-Agent Challenge (SMAC) environment and in a MRCS. Comparative effectiveness analysis with traditional methods in the SMAC environment demonstrates the superior performance of the proposed method. In MRCS, the intelligent countermeasure capability underpinning radar countermeasure is demonstrated, effectively countering strategy formed by typical rule.