SCON: Semantic Cross-Modal Data Association Offset Estimation Network for Radar-Vision Feature Fusion

Unmanned surface vehicles (USVs) play significant roles among various domains. Accurate and robust perception relies on effective information fusion of multiple sensors. The techniques based on fusion of Millimeter-Wave(MMW) radar and cameras performs well in complex environments of water surface, especially in adverse weather conditions. However, the inaccurate height information of MMW measurements, together with the influence of water surface disturbance, results in difficulties in accurately associating radar points with corresponding targets in images. To address these challenges, we propose a network for cross-modal data association offset estimation to improve the fusion of MMW and vision information. It initially conducts column sampling and a category prediction to determine whether radar measurements are associated with target, and subsequently performs offset prediction to align radar points with targets. Additionally, a novel metric named as PTAR (Point Cloud-Target Association Rate) is introduced to quantitatively assess the spatial correlation between radar points and image targets. We validate the effectiveness of the proposed method using a real-world dataset collected on water surface by us. Experimental results demonstrate that the proposed method achieves correct spatial alignment for up to 16% of potential radar points-target pairs.