Adaptive Fusion-Based Two-Flow Spectral–Spatial Transformer Network for SAR-Based Optical Cloud Removal

Synthetic Aperture Radar (SAR) is capable of penetrating cloud cover, providing great potential to handle the challenge of cloud removal when combined with optical remote sensing images. However, existing state-of-the-art techniques are addicted to input channel or feature concatenation strategies, ignoring the exploration of complementary information and leading to unsatisfactory performance. In this study, we propose a simple but effective Adaptive enhanced two-flow Spectral–Spatial Transformer Network (AFSST) for SAR-based optical cloud removal. The proposed network is a two-branch U-shape architecture that leverages the spectral transformer block (SpeTB) to extract global spectral features from optical images and utilizes the spatial transformer blocks (SpaTB) to excavate the spatial attention information from SAR images, respectively. Considering that SAR is often surrounded by noise, yet retains clear edge information, we design a multi-scale SAR feature extraction mechanism (MSFEM) with an adaptive Sobel edge operator to focus on capturing the SAR edge feature. This information is utilized to compensate for the feature mined from the cloudy optical images with learned fusion weights at different feature extraction levels. In addition, we build a dataset for cloud removal in optical images using SAR-assisted data collected from Gaofen-6 and Gaofen-3, to increase the available data and aid in the comparative diversity of newly proposed algorithms. Extensive experiments demonstrate high-quality results of the proposed method, exceeding the performance of state-of-the-art cloud removal approaches. The code and GF-CR dataset will be available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/MinghuaWang123/AFSST-CR</uri>.