Unpaired Spatio-Temporal Fusion for Remote Sensing Images via Deformable Global-Local Feature Alignment

Given a coarse-resolution remote sensing image on a prediction date as input, existing spatio-temporal fusion methods commonly use a pair of coarse and fine resolution images that are acquired close to the prediction date. These images serve as references to predict the corresponding fine-resolution image. Recently, this paradigm has been shifted to be unpaired-reference based that only needs one flexible high-resolution image without date restriction as reference. Despite the flexibility of the reference, current work with this paradigm suffers from the following issue: due to the relatively long revisit period, drastic changes caused by cloud pollution or floods may occur in different periods. It results in large land cover changes of losing texture details and even semantic category shift (e.g., from the land to the water), which makes it difficult to obtain enough high-quality reference data, leading to severe model degradation. To solve the above problems, we propose the deformable global-local feature alignment network (DGFANet) for unpaired spatio-temporal fusion, which combines convolutional neural network and transformer to enhance texture and semantic details through global-local alignment. We design a feature alignment module to link the changed region with the surrounding stable region to obtain the global context information. Next, we perform feature fusion using the cross-communication mixture of experts module, which adaptively retains both local features and global representations. Finally, the color consistency loss is proposed to recover the color change of the fused image. In the experimental portion, DGFANet performs equally or demonstrates superior performance compared to the existing state-of-the-art methods in two widely recognized public datasets Colebly irrigated area and lower Gwydir catchment, which has been shown to do so in experimental studies.