BuildFunc-MoE: An Adaptive Multimodal Mixture-of-Experts Network for Fine-Grained Building Function Identification

Fine-grained building function identification (BFI) is essential for sustainable urban development, land-use analysis, and data-driven spatial planning. Recent progress in fully supervised semantic segmentation has advanced multimodal BFI; however, most approaches still rely on static fusion and lack explicit multi-scale alignment. As a result, they struggle to adaptively integrate heterogeneous inputs and suppress cross-modal interference, which constrains representation learning. To overcome these limitations, we propose BuildFunc-MoE, an adaptive multimodal Mixture-of-Experts (MoE) network built on an effective end-to-end Swin-UNet backbone. The model treats high-resolution remote sensing imagery as the primary input and integrates auxiliary geospatial data such as nighttime light imagery, DEM, and point-of-interest information. An Adaptive Multimodal Fusion Gate (AMMFG) first refines auxiliary features into informative fused representations, which are then combined with the primary modality and passed through multi-scale Swin-MoE blocks that extend standard Swin Transformer blocks with MoE routing. This enables fine-grained, dynamic fusion and alignment between primary and auxiliary modalities across feature scales. BuildFunc-MoE further introduces a Shared Task-Expert Module (STEM), which extends the MoE framework to share experts between the main BFI task and auxiliary tasks (road extraction, green space segmentation, and water body detection), enabling parameter-level transfer. This design enables complementary feature learning, where structural and contextual information jointly enhance the discrimination of building functions, thereby improving identification accuracy while maintaining model compactness. Experiments on the proposed Wuhan-BF multimodal dataset show that, under identical supervision, BuildFunc-MoE outperforms the strongest multimodal baseline by over 2% on average across metrics. Both PyTorch and LuoJiaNET implementations validate its effectiveness, while the latter achieves higher accuracy and faster inference through optimized computation. Overall, BuildFunc-MoE offers a scalable solution for fine-grained BFI with strong potential for urban planning and sustainable governance.