Data Fusion-Based Joint 3D Object Detection Using Point Clouds and Images

Three-dimensional object detection has emerged as a focal point of increasing interest among researchers, driven by advancements in and widespread adoption of autonomous driving technologies. However, this field still faces inherent challenges in single-modal approaches that rely solely on point cloud data for 3D object detection, such as the difficulty in effectively extracting features from sparse point clouds and the lack of critical texture information in the captured representations. To overcome these limitations, we introduce PomageNet, a fusion approach that combines point cloud and image data for 3D object detection. First, initial detection results from the two different kinds of data were applied as inputs, and joint encoding was performed. The encoded joint tensor is then fed into fusion layers. In the fusion stage, multiple 1 × 1 2D convolutions are employed to extract joint high-dimensional features. To enhance feature extraction, a parallel dual-branch framework was designed, and a multidimensional joint encoding mechanism tailored to the network was proposed to better capture contextual information. Experiments show that the capacity of our model is comparable to state-of-the-art (SOTA) methods on KITTI, which was achieved by the proposed network. Results were delivered by the method in detecting small objects, a key challenge in 3D object detection. An average precision (AP) of 67.87% and 60.40% was reached on the cyclist and pedestrian splits of KITTI. Compared to CLOCs, significant improvements were achieved by PomageNet, with 1.28%, 8.40%, and 3.64% increases in the result of detection achieved on the car, cycle, and pedestrian splits of the KITTI dataset.