Beyond One and Two Tower: Cross-Modal Consensus Learning for Image-Text Retrieval

Existing image-text retrieval methods mainly rely on region and word features to measure cross-modal similarities. Thus, dense cross-modal semantic alignment which matches regions and words becomes crucial. However, this is non-trivial due to the heterogeneity gap and the cross-modal attention used to achieve this alignment is inefficient. Towards solving this problem, we propose a novel framework that goes beyond the previous one-tower and two-tower frameworks to learn cross-modal consensus efficiently. The proposed framework does not align regions and words directly like existing methods but uses semantic prototypes as a bridge to attend specific contents with the same semantics among different modalities through semantic decoders, through which cross-modal semantic alignment is naturally achieved. Furthermore, we design a novel plug-and-play self-correction method based on optimal transport to alleviate the drawbacks of incomplete pairwise labels in existing multimodal datasets. On top of various base backbones, we carry out extensive experiments on two benchmark datasets, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i.e</i>., Flickr30K and MS-COCO, demonstrating the effectiveness, superiority and generalization of our method.