Frequency- and Spatial-Domain Saliency Network for Remote Sensing Cross-Modal Retrieval

Recently, remote sensing text-image retrieval has gained increasing attention among researchers due to its capability to provide abundant, inclusive, and multi-perspective information. To extract the salient representation of the two modalities and realize information alignment, existing methods usually apply convolution calculation, matrix multiplication, and attention mechanisms to model features in the spatial domain. However, unlike natural images, remote sensing data contain a considerable amount of noise, and the spatial domain calculation methods will lead to information smoothing and noise exchange, ultimately reducing the robustness of feature representation. In this article, a Frequency and Spatial-domain Saliency Network (FSSN) is proposed by extracting saliency features in both frequency and spatial domains to further enhance the efficiency of retrieval networks. Specifically, the FSSN first designs a Frequency-domain Intra-modal Low-pass Filter (FILF) by using the Fourier Transform to convert the spatial domain representation into the frequency domain representation, and leverage low-pass filtering to filter out the noise information of the image. Afterwards, the Space-domain Inter-modal Boundary-based Saliency module (SIBS) is devised, which fully utilizes the positive and negative boundaries and draws an innovative assessment mechanism to automatically recognize the effective regions and words without noise shifting in the spatial domain. Finally, the Frequency and Spatial-domain Saliency Fusion (FSSF) is designed to realize the effective integration of features obtained from the frequency and spatial domain according to the consistency of the identity masks. Quantitative and qualitative experiments are implemented on four remote sensing benchmarks to showcase the notable effectiveness achieved by the joint modeling network of multiple domains. Specifically, the proposed FSSN outperforms the best model RemoteClip by 7.58% of the mR evaluation matrix on the UCM dataset.