RVF3D: ROI-Driven Vision Transformer Fusion for Multi-Modal 3D Object Detection in Autonomous Vehicles

Current 3D object detection methods face limitations in both single-modal and multimodal approaches. Single-modal detectors struggle with inadequate depth perception or difficulty distinguishing semantically similar objects, while multimodal systems, which combine LiDAR and camera data, encounter challenges like integration complexity and inefficient feature fusion. This study presents ROI-based ViT Fusion (RVF3D), a multimodal 3D object detector combining sparse 3D and dense 2D data, outperforming current methods. The RVF3D enhances the understanding of rich LiDAR and camera representations through improved query generation, feature sampling, and multimodality cross-ViT fusion. We propose an ROI-based Query Sampling (RQS) multimodal 3D object identification pipeline, eliminating laborious non-maximum suppression (NMS) postprocessing and complex prior box configurations. The RQS module uses learnable filters to aggregate image and point representations, retaining foreground characteristics for object localization and recognition while removing background noise. Our RVF3D leverages a hierarchical vision transformer-based approach, ViT-Fusion, which includes CameraViT and LidarViT components for embedding representations of input data in each modality. These representations are fused to enable hierarchical learning and deeper feature extraction. Comprehensive tests on the KITTI and nuScenes datasets demonstrate that RVF3D achieves 89.88% mAP in BEV and 75.2% mAP in 3D, respectively.