Hierarchical Reinforcement Learning for Adaptive Control and Continuous Target Tracking in Cooperative Air Combat Scenarios with Unmanned Wingmen

This study addresses the limited autonomous target-acquisition capabilities exhibited by unmanned wingman systems in beyond-visual-range (BVR) combat scenarios. We propose a hierarchical reinforcement learning framework based on Proximal Policy Optimization (PPO). The dual-layer framework integrates six-degree-of-freedom (6-DoF) flight dynamics with radar lock-on control systems, where the lowlevel controller achieves precise tracking of attitude parameters (heading, airspeed, altitude) with steady-state errors below 1%. Extensive simulations conducted on the JSBSim platform show an 89% success rate in target acquisition across three combat scenarios: random maneuver engagement, head-on interception, and target pursuit. The proposed geometric mean reward formulation with roll angle constraints effectively reduces stall risk. This framework provides an interpretable decision-making architecture for cooperative UAV operations in modern aerial combat systems, overcoming dynamic adaptability constraints in conventional expert systems.