Sparse Deep Interaction Fusion for 3D Object Detection

LiDAR and camera fusion detection significantly improves the accuracy of 3D object detection in autonomous driving. However, the current challenge lies in the fact that multistage fusion methods overly rely on the accuracy of single-modal proposal predictions in the first stage, and the targets missed in the first stage are difficult to compensate in the second stage. On the other hand, fusion methods based on bird’s-eye-view (BEV) directly concatenate different modalities’ information, lacking deep interactions between modalities and struggling to bridge the gap in data distribution. To efficiently fuse rich texture and color information from visual images with accurate spatial information from LiDAR point clouds, we propose SDIFusion (Sparse Deep Interaction Fusion) and design the Sparse Feature Fusion module (SFF) and Aligning Deep Interaction module (ADI). In the SFF module, we utilize LiDAR information to guide the generation of sparse image queries with accurate positions, which are then aligned and fused with dense LiDAR point cloud features. This module achieves a balance between performance and computational cost, providing multi-modal fusion features for obtaining more robust proposals. The ADI module aligns and matches different modal features using shared proposals. It then employs a cross-modal attention mechanism for deep interaction to refine the proposals. Additionally, we propose a user-friendly prior encoding plugin to fuse and enhance the unique advantageous information between different modalities. Finally, we conduct extensive experiments on the nuscenes dataset and achieve a $71.2 \% \mathrm{mAP}$ and 72.9% NDS on the test set, which demonstrates competitive performance and highlights the effectiveness of our approach.