A DRL-Based Path Following and Obstacle Avoidance Method for USV in Water Areas with Environmental Disturbances

This paper presents a deep reinforcement learning method for an unmanned surface vehicle (USV) to follow a desired path and avoid obstacles in the water area containing environmental disturbances (wind and wave). Based on proximal policy optimization (PPO), the improved method learns integrated guidance and heading control policy, which directly maps the USV's navigation states to motor control commands. Firstly, line-of-sight (LOS) guidance law is combined with reward shaping to endow the agent with the capability for path following. Secondly, an additional novel reward function derived from 2D LiDAR data is utilized for end-to-end obstacle avoidance. Moreover, to handle complex environmental disturbances and stabilize the training process, curriculum learning (CL) with multiprocessing is employed to improve the learning speed. Finally, simulations are conducted to verify the effectiveness of the improved PPO compared with LOS+PID and PPO. The results show that the improved method has a lower cross-track error in path following and simultaneously possesses the ability to avoid obstacles.