Enhancing USV Station Keeping via Improved Maximum Entropy Deep Reinforcement Learning

Efficient and accurate station keeping of unmanned surface vehicles (USVs) is essential for applications such as marine geological exploration and unmanned transportation. Traditional methods for USV station keeping are often based on control theory techniques like PID. These methods require manual parameter tuning and rely heavily on expert experience, limiting control autonomy. Recent research has explored the use of deep reinforcement learning (DRL) methods, such as Soft Actor-Critic (SAC), for USV motion control. However, SAC faces challenges such as inefficient exploration and high computational costs, which negatively impact the USV performance of station keeping. To address these issues, we propose an improved SAC algorithm (ISAC) for USV station keeping. First, we introduce a decay factor to regulate the entropy coefficient in the objective function, promoting more efficient exploration during training. Second, we enhance the utilization of USV trajectory data by storing transitions in an experience replay buffer and applying a fair sampling method to improve learning. Experimental evaluations in a high-fidelity simulation environment demonstrate that the ISAC algorithm achieves lower position and orientation errors than several mainstream DRL algorithms, confirming its effectiveness for USV station keeping task.