HDAO: A Hierarchical Curiosity-Driven Reinforcement Learning Approach for AUV Dynamic Obstacle Avoidance

Autonomous obstacle avoidance is a critical capability for Autonomous Underwater Vehicles (AUVs) to operate safely in dynamic and uncertain marine environments. Traditional AUV control methods rely on precise physical modeling and preset rules, yet they struggle to adapt to multiple sources of uncertainty, such as random initial states, dynamic obstacles, and varying currents. In recent years, deep reinforcement learning has provided a new avenue for data-driven adaptive policy learning. However, it remains insufficient for handling long-horizon tasks with sparse rewards. While hierarchical reinforcement learning can mitigate reward sparsity through temporal abstraction, it often faces challenges including exploration–exploitation imbalance, slow global convergence, and insufficient safety guarantees. Furthermore, most existing studies neglect dynamic environmental disturbances and task continuity, which further limits the practical application of these algorithms. To address these challenges, this paper proposes a hierarchical curiosity-driven AUV obstacle avoidance algorithm (HDAO), designed for autonomous obstacle avoidance in dynamic and uncertain underwater environments. The core design of HDAO incorporates several key innovations. Firstly, it introduces a Collision Threat Index for dynamic obstacles, which enables explicit risk perception and quantifies collision threats, thereby enhancing the policy’s generalization and robustness. Secondly, a task-decoupled hierarchical architecture is employed to synergistically optimize global path planning and local obstacle avoidance behaviors. This approach effectively manages long-horizon navigation tasks while alleviating high-dimensional training pressure. Finally, a novel reward mechanism is designed by integrating hierarchical active exploration with curiosity-driven passive exploration. This mechanism effectively incentivizes the agent to explore unvisited areas under sparse reward conditions and dynamically balances exploration and exploitation. Experimental results demonstrate that HDAO significantly outperforms existing methods in terms of obstacle avoidance success rate, training convergence speed and robustness against external disturbances.