3-D Vehicle Detection in Highway Vibration Scenes Using Multimodal Fusion Network With Point-Voxel Global-Align

In highway scenarios, LiDAR and cameras are prone to vibrations due to the rapid passage of vehicles through gantries and environmental factors. To address the decline in detection accuracy caused by this vibration, this study investigates the use of hard projection connections and adaptive soft attention-based connections for voxel- and point-level feature fusion. First, to maintain consistency in data augmentation between point clouds and images, a cross-modal GT-AUG is constructed, employing a cross-modal feature alignment data augmentation method applied to both point clouds and images simultaneously. Next, to link LiDAR bird’s eye view (BEV) with image features, this article introduces a cross-modal multihead attention mechanism, utilizing a LiDAR BEV-guided query initialization strategy to select object queries based on LiDAR BEV information. Finally, the point-voxel fusion (PVF) with global-align module (GAM) is developed, employing the GAM structure for point-image and voxel-image feature aggregation. Extensive comparative experiments conducted on the nuScenes and nuScenes-C mixed datasets validate the model’s effectiveness. The results demonstrate that the proposed model successfully addresses the feature misalignment issue in scenes, while image data compensates for the semantic information corresponding to sparse point cloud locations, achieving mean average precision (mAP) and nuScenes detection score (NDS) scores of 62.25% and 68.23% on the mixed validation set, respectively.