An Experience Replay Approach Based on SSIM to Solve the Sparse Reward Problem in Pursuit Evasion Game^*

Reward sparsity has always been a challenge in the field of reinforcement learning. Because of the complexity, dynamics, and uncertainty of the state space of the pursuit-evasion game, the problem of reward sparsity faced by agents in the learning process is more prominent. To solve this problem, we proposed an empirical replay method using the structural similarity measure (SSIM). The approach assesses the novelty of the current pursuit-evasion game screen, encouraging the agent to prioritize learning from states that appear less frequently but have greater learning potential. Simulation results illustrate that our approach outperforms both random sampling experience replay and prioritized experience replay methods in terms of training effectiveness and successful pursuit time within the pursuit-evasion game environment.