Contextual Relation Guided Image-Text Matching

As a fundamental branch of cross-modal retrieval, the challenge of mitigating the disparities inherent in image and text modalities in image-text matching continues. While existing endeavours have narrowed the semantic gap, most of them overlook the intricate semantic relations among objects. With the aim of addressing this issue, we propose a Contextual Relation Guided Image-Text Matching Network (CRGN), which explores global information, fine-grained local information and contextual relationships to perform multi-granularity alignment. Primarily, we extract both global and local image features to capture the overall and fine-grained semantic information of images. Subsequently, a visual context coding module is designed to mine the relationships among local image features. Concurrently, for the text modality, we use a backbone network to extract local features that capture contextual information, and then, we obtain global features using the self-attention mechanism. Finally, we employ a multi-granularity cross-attention network to simultaneously align the global features and local features of the image and text to capture multi-granularity semantic information. The proposed method demonstrates significant performance on the FLICKR30K and MSCOCO datasets, as shown by the experimental results.