Multi-modal Camera and One-beam LiDAR Fusion Network for Low-cost 3D Object Detection

3D object detection is a fundamental technique for scene perception of autonomous robots. Recent methods rely heavily on muti-beam LiDAR such as 64-beam, which is limited by the cost and power in practical applications. Monocular camera or one-beam LiDAR provides optional ways for this problem. However, the front one is inherently ill-posed in depth prediction and the latter one could not well infer semantics, size and pose of 3D objects. In addition, the fusion of camera and one-beam LiDAR is not well studied in recent methods. We specifically investigate a framework of 3D object detection by fusing monocular camera and one-beam LiDAR. Different from recent multi-modal techniques, which primarily rely on LiDAR data, our solution mainly utilizes camera to detect 3D objects and one-beam LiDAR as a supplement. First, a high-quality depth map is predicted from an RGB image and one-beam LiDAR points with a two-branch network and a global-local normal gradient loss. Second, pseudo-LiDAR points are converted from the depth map and further corrected by the one-beam LiDAR points. Finally, corrected pseudo-LiDAR points are fed into a mature LiDAR-based 3D detector. Our method is verified on KITTI test set and compared to twelve recent baselines of LiDAR-based, RGB-based and multi-modal methods. Extensive results verify that our method achieves 38.00, 27.36 and 22.92 of AP40 <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\text{IoU}=0.7)$</tex> in easy, moderate and hard modes, which significantly outperforms all the baselines.