Buoy and Winch Collaborative Control System Based on Deep Reinforcement Learning

The improved control performance of the buoy and winch collaborative control system can enhance the stability of the connection between underwater robots and ground industrial control equipment. To overcome the challenge of mathematical modeling of this control system, this research introduces reinforcement learning and transformer models in the design process. The main contributions include the development of two simulation environments for training DRL agents, designing a reward function to guide the exploration process, proposing a buoy control algorithm based on the discrete soft actor-critic (SAC) algorithm, and proposing a winch cable length prediction algorithm based on a lightweight transformer model. The experiment results demonstrated significant improvements in rewards diagrams, buoy control trajectories, and winch model performance, showcasing the effectiveness of our proposed system. The average error of the buoy tracking trajectories induced by different policies trained in the two environments is less than 0.05, and the evaluation error of the behavior cloning lightweight transformer model is less than 0.03.