Multistage Fusion Object Detection With Marine Radar and Camera in Complex Maritime Context

Marine object detection is an essential component for maritime autonomous surface ship (MASS) systems, enabling them to perceive obstacle objects and avoid them. Existing detection algorithms primarily rely on visual sensors and face serious limitations in complex maritime environments. In this paper, we propose a novel joint object detection method based on the fusion of marine radar and camera data for robust maritime object detection. By employing a multi-stage hybrid-level fusion architecture, the proposed method efficiently fuses marine radar data with camera images, thereby enhancing the accuracy and robustness of object detection in complex maritime contexts. In addition, inspired by vision transformer detectors, a new transformer-based fusion module is proposed to mitigate semantic disparity between the two modalities and enhance radar information guidance through extracted object queries. Extensive experiments on public maritime perception datasets demonstrate the significant advantages of the proposed method in various maritime environments. The framework also exhibits strong generalization capability, making it applicable to diverse maritime settings, including challenging conditions like low visibility and high background noise. In public maritime perception datasets, the proposed method achieves state-of-the-art accuracy compared to other single-modality baselines and radar-camera fusion baselines, outperforming the excellent visual object detection model YOLOv8-x in terms of mAP0.35 increased by 13.72% in common scenarios and 24.85% in complex scenarios.