NCFDet: Enhanced point cloud features using the neural collapse phenomenon in multimodal fusion for 3D object detection

Abstract The accurate and effective detection of 3D objects represents a crucial component in the development of multi-sensor autonomous driving and robotics systems, particularly in the context of navigating complex urban environments. The complementary nature of image and point cloud data allows for the detection of objects with greater accuracy and robustness when both image and point cloud features are employed. At present, there is no optimal solution for the timing of multi-sensor fusion, particularly in the case of cross-modal data formats. In order to address these issues, we propose a multi-sensor object detection scheme based on Neural Collapse (NC) theory augmented point cloud, which we have designated NCFDet. In particular, we have incorporated an image pre-training model at the network layer where the NC occurs and designed a fusion module based on Transformer Attention. Furthermore, we investigate the interconnection between NC and transmodal transfer, and provide an explanation for the efficacy of the former based on the latter. The NCFDet system performs well for the detection of small targets in scenes due to the advantage of a compact image data format. The performance of the proposed NCFDet system was validated on the KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) dataset in comparison to existing methods. Furthermore, the object detection performance of the proposed framework was demonstrated on our visual rotation platform. The results show that the detection system achieves advanced fusion results. The code will be published after the paper is published.