Semantic segmentation of single SAR imagery leveraging historical Sentinel-1 and Sentinel-2 data

Optical and Synthetic Aperture Radar (SAR) images are widely used in land cover mapping, environmental monitoring, and disaster management. While optical sensors are often hindered by adverse weather conditions such as clouds and rain, SAR provides all-weather capability but suffers from lower interpretability. To overcome this limitation, historical optical data can be leveraged to improve SAR interpretation and enhance mapping accuracy. This study proposed a degradable multimodal fusion (DEMFuse) framework that leverages historical optical data to enhance the interpretation of single SAR imagery and implemented it via semantic segmentation. The DEMFuse is composed of a SAR-to-optical image generator and a degradable fusion model. The former was built upon an ImageNet-pretrained transformer that employed a non-local generative modeling for translating SAR images into optical images to recover optical visual structural information. The latter was proposed to achieve progressively enhanced fusion performance by continuously assimilating useful information from synthesized optical data based on SAR data, thus reducing the impact of artifacts in the synthesized data, and ensuring the effective fusion of the optical and SAR data. To demonstrate the effectiveness of the proposed DEMFuse framework, a globally distributed SAR-to-optical image translation dataset and a land-cover semantic segmentation dataset were constructed by using 18,071 Sentinel-1 SAR and Sentinel-2 optical images, together referred to as “IT-SS-18 K” (image translation-segmentation-18 K). Additionally, a SAR-based flood rapid mapping dataset was employed to validate the effectiveness of the proposed framework in disaster scenarios. Experiments on IT-SS-18 K demonstrated that the proposed DEMFuse framework significantly reduced the relative accuracy gap by over 30 % between SAR and optical data for the segmentation of water, built areas, and trees. In the flood disaster scenario, the proposed framework yielded a 1.76 % improvement compared to the single SAR segmentation. This finding suggests that DEMFuse can be employed to enhance SAR interpretation in scenarios where optical data is missing, and improve the efficiency and accuracy of disaster management and environmental monitoring. The datasets will be made available at https://github.com/RCAIG/DEMFuse.