SegCR: A Multimodal and Multitask Complementary Fusion Network for Remote Sensing Semantic Segmentation and Cloud Removal

Synthetic Aperture Radar (SAR) can provide complementary information to optical images due to its insensitivity to atmospheric conditions, making optical-SAR fusion semantic segmentation a popular research topic. However, existing optical-SAR fusion methods overlook cloud interference in real-world scenarios, leading to suboptimal performance in practical applications. To address the performance degradation of existing optical-SAR fusion semantic segmentation methods under cloud interference, we propose SegCR, a multimodal and multitask framework that leverages the complementary information of SAR images to reduce the negative influence of clouds in optical images. Specifically, we extract a cloud impact map using the frequency differences between optical and SAR features, which represents the extent of cloud interference for every pixel in the optical image. Next, we introduce a SAR-to-OPT translation sub-task, leveraging SAR features to produce optical simulated features that supplement the missing information in the cloud-affected optical features. Then, based on the cloud impact map, we perform a targeted complementary fusion of optical and SAR features. Finally, we design a multitask learning framework that simultaneously performs semantic segmentation and cloud removal, enabling the high-level semantic understanding task and the low-level vision task to enhance each other through multitask learning. Extensive comparison experiments on two publicly multitask remote sensing datasets reveal that our proposed SegCR outperforms the existing state-of-the-art methods. Additionally, ablation experiments have confirmed the effectiveness of the proposed module and the multitask learning framework.