Transformer-Based LiDAR-Camera Fusion for Semantic Segmentation

In the realm of autonomous driving, precise semantic segmentation of LiDAR point clouds is crucial for robust scene understanding. Current methods often struggle with feature extraction and fusion of LiDAR and camera data. This paper proposes a novel transformer-based semantic segmentation network that effectively integrates LiDAR point clouds with camera images. By leveraging a multi-head self-attention mechanism, our approach extracts features from both modalities, utilizing cubic and radial windows in the LiDAR stream and overlapping window region coding in the camera stream. Through multi-scale knowledge refinement and knowledge distillation, we successfully integrate image prior knowledge into the LiDAR stream, thereby significantly improving segmentation accuracy while maintaining a high inference speed. Experimental results on the SemanticKITTI and nuScenes datasets demonstrate the superiority of our method, achieving mIoU scores of 75.4% and 80.6%, respectively, with an inference speed of 82 ms per frame. Our work not only advances the state-of-the-art in LiDAR point cloud semantic segmentation, but also highlights the importance of integrating imaging features and transformer-based knowledge distillation in this field.