Noise-driven enhancement for exploration: Deep reinforcement learning for UAV autonomous navigation in complex environments

Unmanned Aerial Vehicle (UAV) plays a prominent role in various fields, and autonomous navigation is a crucial component of UAV intelligence. Deep Reinforcement Learning (DRL) has expanded the research avenues for addressing challenges in autonomous navigation. Nonetheless, challenges persist, including getting stuck in local optima, consuming excessive computations during action space exploration, and neglecting deterministic experience. This paper proposes a noise-driven enhancement strategy. In accordance with the overall learning phases, a global noise control method is designed, while a differentiated local noise control method is developed by analyzing the exploration demands of four typical situations encountered by UAV during navigation. Both methods are integrated into a dual-model for noise control to regulate action space exploration. Furthermore, noise dual experience replay buffers are designed to optimize the rational utilization of both deterministic and noisy experience. In uncertain environments, based on the Twin Delay Deep Deterministic Policy Gradient (TD3) algorithm with Long Short-Term Memory (LSTM) network and Priority Experience Replay (PER), a Noise-Driven Enhancement Priority Memory TD3 (NDE-PMTD3) is developed. We established a simulation environment to compare different algorithms, and the performance of the algorithms is analyzed in various scenarios. The training results indicate that the proposed algorithm accelerates the convergence speed and enhances the convergence stability. In test experiments, the proposed algorithm successfully and efficiently performs autonomous navigation tasks in diverse environments, demonstrating superior generalization results.