A Multiscale Hybrid Model for Natural and Remote Sensing Image-Text Retrieval

Image-text retrieval is an essential task in multimodal information processing. Recently, deep learning methods have achieved significant progress in cross-modal image-text retrieval by learning both coarse-grained representations of entire images and texts, as well as fine-grained representations of image regions and words, enabling semantic alignment at global and local levels. Drawing from multi-scale strategies often used in remote sensing image retrieval, this paper introduces multi-scale features into the framework as medium-grained image representations. We propose a model, MHL, which integrates three levels of granularity: global, local, and multi-scale. MHL extracts features from images and texts using Faster-RCNN, BERT, and a multi-scale feature fusion module, and computes three types of similarity. Our framework uses a three-stage inference approach: first, initial retrieval based on global similarity to identify candidate samples, followed by further refinement using additional similarity measures. Extensive experiments on the Flickr30K and RSICD datasets show that MHL outperforms state-of-the-art methods, demonstrating superior retrieval performance and efficiency for both natural and remote sensing images.