Deep reinforcement learning from human preferences for ROV path tracking

Remotely operated vehicles (ROVs) play an indispensable role in the ongoing exploration and utilization of ocean resources due to their flexibility. Deep reinforcement learning has been used to increase the autonomy of ROV operations, so as to reduce the workload on operators and minimize human errors in operations. However, it is challenging to design a reasonable reward function for various tasks due to the complexity and uncertainty of the actual marine environment. In this paper, we propose and implement a Preference-based Reinforcement Learning (PbRL) method for ROV path tracking, which can learn control policies from a trained reward predictor based on human preferences over provided segments of ROV’s trajectories. We evaluated our proposed PbRL method in three tasks: 2D straight line path tracking, 2D sinusoidal curve path tracking, and 3D straight line path tracking, using the BlueROV2 simulator on the Gazebo platform. Our results show consistent significant advantage of our PbRL method learning from human preferences over traditional reinforcement learning PPO algorithm from predefined reward function and GAIL algorithm learning from demonstrated trajectories. • Proposed a Preference-based Reinforcement Learning (PbRL) for ROV path tracking • Introduced the concept agent training through human preferences • Results with ROV simulator show consistent advantages of PbRL over PPO method