Diverse strategy generation for orbital pursuit-evasion games using distributed reinforcement learning

• A discrete-continuous hybrid action space network is developed to generate game strategies adaptable to diverse control modes in complex orbital engagements. • A combination of intrinsic and extrinsic rewards is employed to achieve hybrid constrained objectives, facilitating multi-objective task execution in realistic orbital game scenarios. • A distributed population-based iterative technique is proposed to generate a diverse set of strategies, enabling realistic game behavior under unknown opponent policies. Under the complex orbital environment with multi-objective constraints, aimed to solve the game control of impulsive maneuver spacecrafts, this paper proposes an adaptable distributed reinforcement learning-based method to generate diverse impulsive orbital pursuit-evasion game (OPEG) strategies. At the outset, a high-fidelity simulation system is developed to model the OPEG, considering multiple orbital perturbations, along with the game scenario framework to describe multi-objective constraints and terminal adjudication. To devise a multi-objective solution, we extracting multi-dimensional features from environmental observations at first, while intrinsic and extrinsic rewards are combined to cater to different game objectives. To enhance flexibility in handling complex maneuver-based orbital game scenarios, a hybrid action space decision model based on deep neural networks is subsequently developed. The model generates discrete actions to select maneuver directions and continuous actions to compute the corresponding velocity increments. On this basis, the decision model is trained using a distributed deep reinforcement learning algorithm. A winning pool mechanism based on win rate correlation is introduced to enhance training effectiveness, while an asymmetric population strategy augmentation technique is employed for parallel iterative training, improving the diversity and robustness of the population strategies. Experimental results validated in a one-on-one OPEG simulation environment with random initial positions in GEO, yielding interpretable pursuit-evasion maneuvering strategies trained via end-to-end neural networks that closely resemble real-world orbital game scenarios. The proposed method generates diverse strategy populations satisfying Nash equilibrium for both pursuers and evaders, providing strategic support for spacecraft maneuvering in orbital space games.