A novel projection map driven multimodal fusion framework for ALS point cloud semantic segmentation

Semantic segmentation of urban point clouds captured by Airborne Laser Scanning (ALS) is essential for understanding complex 3D environments, serving as a robust underlying data foundation for digital twin applications. The fusion of multimodal data has been proven to significantly improve the performance of ALS semantic segmentation by fully mining rich complementary information in each modality. However, existing fusion-based ALS semantic segmentation methods face critical limitations due to the reliance on multiple sensors, which constrains their applicability. To this end, we propose a novel multimodal framework E levation G uidance A daptive F used Network, termed EGAFNet, that integrates naturally formed top-view projection images from ALS to enhance the information perception of the point cloud. The framework focuses on utilizing projection images, structured around two key components: input representation and feature representation. Specifically, to generate highly discriminative input representation, we propose a novel projection method that accurately preserves the relative height relationships between objects and develop a Height Adaptive Scaling Module (HASM) to adaptively adjust object heights, enhancing the expressive capability of elevation information in the projection images. As for feature representation, we design a dual-branch network that effectively captures local and global context from the projection images within a large receptive field. Meanwhile, we propose an Elevation Guidance Adaptive Fusion Module (EGAFM) that adaptively fuses 2D and 3D features based on occlusion relationships to reduce feature confusion caused by occlusion in elevation projection, ensuring meaningful fusion between multimodal features. Extensive experiments on three public datasets demonstrate that our EGAFNet outperforms current state-of-the-art methods. • A multimodal network EGAFNet for urban-scale point cloud semantic segmentation. • A discriminative z-aware image via terrain variation elimination and HASM. • Tailor-designed large-scale local and global context for enhanced 3D feature learning. • A fusion module EGAFM effectively integrates 2D and 3D features considering occlusion. • Extensive experiments demonstrate the robustness and superiority of EGAFNet.