Semantics-Fusion: Radar Semantic Information-Based Radar–Camera Fusion for 3D Object Detection

The fusion of millimeter-wave radar and camera for three-dimensional (3D) object detection represents a pivotal technology for autonomous driving, yet it is not without its inherent challenges. First, the radar point cloud contains clutter, which can result in the generation of impure radar features. Second, the radar point cloud is sparse, which presents a challenge in fully extracting the radar features. This can result in the loss of object information, leading to object misdetection, omission, and a reduction in the robustness. To address these issues, a 3D object detection method based on the semantic information of radar features and camera fusion (Semantics-Fusion) is proposed. Initially, the image features are extracted through the centroid detection network, resulting in the generation of a preliminary 3D bounding box for the objects. Subsequently, the radar point cloud is clustered based on the objects’ position and velocity, thereby eliminating irrelevant point cloud and clutter. The clustered radar point cloud is projected onto the image plane, thereby forming a radar 2D pseudo-image. This is then input to the designed 2D convolution module, which enables the full extraction of the semantic information of the radar features. Ultimately, the radar features are fused with the image features, and secondary regression is employed to achieve robust 3D object detection. The performance of our method was evaluated on the nuScenes dataset, achieving a mean average precision (mAP) of 0.325 and a nuScenes detection score (NDS) of 0.462.