CLIP-based semantic refinement method for image-text retrieval

Abstract Although existing image-text retrieval methods show strong retrieval performance, they generally fail to explicitly model the subtle semantic difference within categories. To address this problem, we propose a new image-text retrieval method, called CLIP-based Semantic Refinement Method For Image-Text Retrieval (CLIP2SRITR). The proposed method consists of a modal alignment part and a semantic matching part, where the former is to learn better alignment between two modalities and the latter is to enhance the aggregation of image-text features intra-class. The pipeline helps model the distinction between image-text pairs with semantic information differences. In addition, CLIP2SRITR leverages additive angular margin for image-text retrieval loss function to maximize classification boundaries, achieving the goal of increasing intra-class similarity and inter-class variability. Through extensive experiments on three image-text benchmarks, e.g. Wikipedia, Pascal-Sentence, and NUS-WIDE, we show that CLIP2SRITR outperforms state-of-the-art methods, verifying the effectiveness of our method.