Simultaneous Depth and Heading Control for Autonomous Underwater Vehicle Docking Maneuvers Using Deep Reinforcement Learning within a Digital Twin System

This study proposes an automatic control system for Autonomous Underwater Vehicle (AUV) docking, utilizing a digital twin (DT) environment based on the HoloOcean platform, which integrates six-degree-of-freedom (6-DOF) motion equations and hydrodynamic coefficients to create a realistic simulation. Although conventional model-based and visual servoing approaches often struggle in dynamic underwater environments due to limited adaptability and extensive parameter tuning requirements, deep reinforcement learning (DRL) offers a promising alternative. In the positioning stage, the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm is employed for synchronized depth and heading control, which offers stable training, reduced overestimation bias, and superior handling of continuous control compared to other DRL methods. During the searching stage, zig-zag heading motion combined with a state-of-the-art object detection algorithm facilitates docking station localization. For the docking stage, this study proposes an innovative Image-based DDPG (I-DDPG), enhanced and trained in a Unity-MATLAB simulation environment, to achieve visual target tracking. Furthermore, integrating a DT environment enables efficient and safe policy training, reduces dependence on costly real-world tests, and improves sim-to-real transfer performance. Both simulation and real-world experiments were conducted, demonstrating the effectiveness of the system in improving AUV control strategies and supporting the transition from simulation to real-world operations in underwater environments. The results highlight the scalability and robustness of the proposed system, as evidenced by the TD3 controller achieving 25% less oscillation than the adaptive fuzzy controller when reaching the target depth, thereby demonstrating superior stability, accuracy, and potential for broader and more complex autonomous underwater tasks.