Implicit Fine-Grained Alignment for Cross-Modal Retrieval

Cross-modal retrieval has always been one of the essential tasks in multimodal learning, with the aim of searching for the most relevant cross-modal samples based on a given query. Taking image-text retrieval as an example, a direct and efficient approach is to use visual semantic embedding technology to bring positive samples closer and push away negative samples in the shared subspace. However, these methods often overlook the potential implicit alignment relationships in text. In this letter, we enhance representation by revisiting potential alignment relationships in text and propose a new method called IACR that exploits an Implicit Alignment Module (IAM), thereby capturing the fine-grained local associations. Specifically, IAM captures implicit alignment through a multi-head attention mechanism to establish fine-grained alignment between image representations and partial words in the text. This approach avoids the additional reasoning costs caused by fine-grained modeling and maintains local correlation to improve performance. To verify the superiority of this cross-modal alignment paradigm, we conduct experiments on two large-scale image-text retrieval datasets, i.e., Flickr30K and MS-COCO, to evaluate the proposed IACR and provide analysis to verify its effectiveness.