Urban informal settlements interpretation via a novel multi-modal Kolmogorov–Arnold fusion network by exploring hierarchical features from remote sensing and street view images

Urban informal settlements (UIS) interpretation has important scientific value for achieving urban sustainable development. Recent research on UIS interpretation tasks mainly includes the single-modality method, which uses remote sensing images, and the multi-modality method which uses remote sensing and geospatial data. However, from a single remote sensing perspective, the inter-class similarities, and a regional mixture of complex geo-objects from a bird-eye perspective of UIS areas make UIS interpretation extremely challenging. The current multi-modal methods cannot fully explore the modality-specific features within the modality or ignore the modality-correlation features between different modalities. To address these issues, this study proposed a novel multi-modal Kolmogorov–Arnold fusion network, namely KANFusion, to explore the modality-specific features within the modality and fuse the modality-correlation features between different modalities to boost UIS interpretation using remote sensing and street view images. The proposed KANFusion model employs the Kolmogorov–Arnold Network (KAN) instead of the conventional MLP structure to enhance the model-fitting capability of heterogeneous modality-specific features and uses a novel Multi-level Feature Fusion Module with KAN block (MFF) to fuse the hierarchical modality-specific and modality-fusion features from remote sensing and street view images for better UIS interpretation performance. We conducted extensive experiments on the manually annotated ChinaUIS dataset of eight megacities in China and a public S 2 U V dataset and compared the proposed KANFusion with other state-of-the-art methods. The experimental results confirmed the superiority of the proposed KANFusion. This work is available in https://github.com/cyg-nhyang/KANFusion . • A ChinaUIS dataset containing 1833 manually labeled samples of UIS is constructed. • A multi-modal Kolmogorov–Arnold fusion network is proposed to fuse features. • The Kolmogorov–Arnold Network we employed enhances the feature learning capability. • Fusing RSI and SVI provides an applicable way of improving UIS Interpretation.