Curiosity-Driven Distributional Soft Actor-Critic for AUV Anti-Disturbance Path Tracking

With the advancement of marine resource exploration and exploitation technologies, autonomous underwater vehicles (AUVs) have shown significant potential to perform underwater tasks, such as pipeline maintenance. However, traditional control algorithms, such as proportional-integral-derivative, sliding mode, and model predictive control, struggle to adapt to nonlinear and current-disturbed underwater environments, which impedes their accuracy and stability in path-tracking tasks. To address these challenges, this paper proposes a curiosity-driven distributional soft actor-critic framework. The framework leverages distributional soft actor-critic algorithms to control the navigation direction of the AUV and employs traditional proportional-integral control to maintain navigation speed, creating a stable, high-precision control strategy for complex underwater environments. Building on this framework, this paper further advances its capabilities through two key improvements. First, a curiosity-driven automatic entropy adjustment technique is designed to enhance the exploration of unknown states and the utilization of similar states, thus optimizing the accuracy of path tracking. Second, a dynamic prioritized experience replay mechanism is developed to prioritize learning samples based on time-differential errors and historical rewards, thereby improving learning efficiency and stability. Simulation experiments are conducted in two different underwater environments with and without ocean currents. Compared with reinforcement learning-based methods and traditional control algorithms, the proposed method has significant advantages in terms of accuracy, stability, and adaptability.