Acoustic and Visual Information Fusion of Unmanned Surface Vehicles

In many scenarios, sensors installed in USV scan only detect either the above-water or underwater part of ob-stacles, which leads to errors in obstacle avoidance. In this paper, the obstacle information detected by the sonar and visual camera is used to perform cross-domain data fusion. In low-speed operation mode, the prior binding information (surface and underwater) such as coordinate and direction will be used to match the collected image information (surface and underwater) until the desired matching level is achieved, directly determining the target orientation. When the coordinate matching level falls below the threshold or when the target ID information is missing, data processing module including cluster analysis, confidence filtering, and information entropy assessment will be employed to allocate weights to heterogeneous information, such as the distance and direction of the obstacle. In the fusion model, the Monte Carlo algorithm is applied to calculate the nonsynchronous error between the obstacles detected by the sonar and the visual sensor. Corresponding angle calculations are performed, and the information such as the distance and direction is converted into latitude and longitude coordinates. Compared to the traditional approach of simply merging surface or underwater obstacle information, we have integrated the information of the same obstacle from both above and below the water surface and enhance the reliability and accuracy of decision-making for unmanned surface vehicles.