Multi-USV Formation Collision Avoidance via Deep Reinforcement Learning and COLREGs

Dear Editor, This letter focuses on the collision avoidance for a multi-unmanned surface vehicle (multi-USV) system. A novel multi-USV collision avoidance (MUCA) algorithm is proposed. Firstly, in order to get a more reasonable collision avoidance policy, reward functions are constructed according to international regulations for preventing col-lisions at sea (COLREGS) and USV dynamics. Secondly, to reduce data noises and the impacts of outliers, an improved normalization method is proposed. States and rewards of USVs are normalized to avoid gradient vanishing and exploding. Thirdly, a novel <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\epsilon$</tex>-greedy method is proposed to help the optimal policy converge faster. It is easier for USVs to explore the optimal policy in the learning process. Finally, the proposed MUCA algorithm is tested in a multi-encounter situation including head-on, crossing, and overtaking. The experimental results demonstrate that the newly proposed MUCA algorithm can provide a collision-free marching policy for the USVs in formation.