Sparse Sensor Fusion for 3D Object Detection with Symmetry-Aware Colored Point Clouds

Multimodal fusion-based object detection is the foundational sensing task in scene understanding. It capitalizes on LiDAR and camera data to boost the robust results. However, there are still great challenges in establishing an effective fusion mechanism and performing accurate and diverse feature interaction fusion. In particular, the relationship construction between the two modalities has not been comprehensively exploited, leading to sensor data utilization deficiencies and redundancies. In this paper, a novel 3D object-detection framework, namely a symmetry-aware sparse sensor fusion detection network (2SFNet), is proposed. This framework was designed to leverage point clouds and RGB images. The 2SFNet consists of three submodules, filtered colored point cloud generation, pseudo-image generation, and a dilated feature fusion network, to solve these problems. Firstly, filtered colored point cloud generation constructs non-ground colored point cloud (NCPC) data by employing an early fusion strategy and a ground-height-filtering module, selectively retaining only object-related information. Subsequently, 2D grid encoding is used on the reduced colored data. Finally, the processed colored data are fed into the improved PillarsNet architecture, which now has expanded receptive fields to enhance the fusion effect. This design optimizes the fusion process by ensuring a more balanced and effective data representation, aligning with the symmetry concept that underlies the model’s functionality. Experiments and evaluations were conducted on the KITTI dataset to present the effectuality, particularly for categories characterized by sparse point clouds. The results indicate that the symmetry-aware design of the 2SFNet leads to an improved performance when compared to other multimodal fusion networks, and alleviates the phenomenon caused by highly obscured and crowded scenes.