3DMMF: 3D object detection network based on multi-layer and multi-modal fusion

Abstract Cameras and LiDAR are important sensors in autonomous driving systems that can provide complementary information to each other. However, most of the LiDAR-only methods outperform the fusion method on the main benchmark datasets. Current studies attribute the reasons to the misalignment of views and difficulty in matching heterogeneous features. In particular, the single-stage fusion method makes it difficult to fully fuse the features of the image and point cloud. In this work, we propose 3DMMF: a 3D object detection network based on multi-layer and multi-modal fusion methods. 3DMMF works by painting and encoding point clouds in the frustum proposed by the 2D object detection network. Then the painted point cloud is fed to the LiDAR-only object detection network which has expanded channels and a self-attention mechanism module. Finally, CLOCs are used to match the geometric direction features and category semantic features of 2D and 3D detection results. Experiments on KITTI datasets show that this fusion method has a significant improvement over the baseline of the LiDAR-only method, with an average mAP improvement of 6.3%.