FP3Seg: Point Cloud Panoptic Segmentation via LiDAR-Camera Fusion and Progressive Decoder

Point cloud panoptic segmentation is a 3D scene perception task that provides a holistic solution for both semantic and instance segmentation. The sparsity and lack of texture features in LiDAR point cloud, coupled with the relatively narrow field of view of camera, make multi-modal fusion challenging. In this paper, we propose a novel multi-modal fusion based point cloud panoptic segmentation method, named FP3Seg, with main contributions including Hybrid Domain Adaptive Fusion (HDAF) module and Progressive Decoder. HDAF employs learnable weights to adaptively fuse multi-modal features in both the channel and spatial domains. Through knowledge distillation, FP3Seg extends the benefits of multi-modal fusion beyond the camera field of view. Pro-gressive Decoder embeds semantic and instance information into the input of panoptic decoder, assisting the decoder in understanding the distinctions between stuff and thing classes. The proposed method is benchmarked on the SemanticKITTI test set, achieving 57.7% PQ, showing a 1.7% improvement over baseline. Experimental results demonstrate that FP3Seg possesses advantages over single-modal approaches in multiple aspects, especially for the segmentation of thing classes.