Surface Vessels Detection and Tracking Method and Datasets with Multi-Source Data Fusion in Real-World Complex Scenarios

Environment sensing plays an important role for the safe autonomous navigation of intelligent ships. However, the inherent limitations of sensors, such as the low frequency of the automatic identification system (AIS), blind zone of the marine radar, and lack of depth information in visible images, make it difficult to achieve accurate sensing with a single modality of sensor data. To overcome this limitation, we propose a new multi-source data fusion framework and technologies that integrate AIS, radar, and visible data. This framework leverages the complementary strengths of these different types of sensors to enhance sensing performance, especially in real complex scenarios where single-modality data are significantly affected by blind zone and adverse weather conditions. We first design a multi-stage detection and tracking method (named MSTrack). By feeding the historical fusion results back to earlier tracking stages, the proposed method identifies and refines potential missing detections from the layered detection and tracking processes of radar and visible images. Then, a cascade association matching method is proposed to realize the association between multi-source trajectories. It first performs pairwise association in a high-accuracy aligned coordinate system, followed by association in a low-accuracy coordinate system and integrated matching between multi-source data. Through these association operations, the method can effectively reduce the association errors caused by measurement noise and projection system errors. Furthermore, we develop the first multi-source fusion dataset for intelligent vessel (WHUT-MSFVessel), and validate our methods. The experimental results show that our multi-source data fusion methods significantly improve the sensing accuracy and identity consistency of tracking, achieving average MOTA scores of 0.872 and 0.938 on the radar and visible images, respectively, and IDF1 scores of 0.811 and 0.929. Additionally, the fusion accuracy reaches up to 0.9, which can provide vessels with a comprehensive perception of the navigation environment for safer navigation.