Reinforcement Learning based Hovering Control of a Buoyancy Driven Unmanned Underwater Vehicle with Discrete Inputs

This paper focuses on the hovering control problem of large displacement unmanned underwater vehicles (LDUUV). The dynamic model exhibits significant uncertainty with the numerous devices carried by LDUUV. In addition, the variable buoyancy system (VBS) of discrete control inputs and timevarying characteristics, makes it difficult for traditional modelbased continuous controllers to address these challenges effectively. A discrete proximal policy optimization (PPO) of the hovering control algorithm based on model-free reinforcement learning (RL) is proposed, which provides a new solution for the hovering control of LDUUVs with reduced frequent switching of control inputs. First, the hovering control problem of LDUUVs is modelled as a Markov Decision Process (MDP) with continuous states and discrete actions. Second, the importance sampling technique is introduced in the strategy improvement stage, combined with the PPO algorithm’s clipped objective function. The policy network maps the discrete actions output to actual control instructions and ensures the stability of policy updates through clipping objective functions and Kullback-Leibler (KL) divergence constraints. Additionally, the VBS regulates the buoyancy of the LDUUV based on these discrete motion outputs, enabling precise vertical-plane hovering. Third, this paper employs trust region policy optimization (TRPO) as the final policy optimization method, enforcing KL divergence constraints to ensure efficient optimization while maintaining policy smoothness. Simulation results demonstrate that the proposed method significantly enhances the hovering accuracy and stability of the LDUUV in the navigation process.