Selective Multi-grained Alignment for Text-Video Retrieval

Multi-grained alignment for text-video retrieval takes advantage of both coarse- and fine-grained semantic alignments, which can enhance the representation capabilities of cross-modal information retrieval. However, the current research fails to fully explore the informative semantics embedded in fine-grained levels of both text and video spaces. In this work, we propose a Selective Multi-Grained Alignment framework, termed SMA, which achieves video-sentence and object-phrase alignments through coarse- and fine-grained alignment modules. Specifically, we introduce a token aggregation module, which implicitly selects important patches per frame and words in text and then aggregates them to generate object-level features for frames and phrase-level features for text. Moreover, we apply a similarity-aware selection module to select keyframes in video efficiently. Extensive experiments on four benchmarks demonstrate that our SMA achieves superior performance on MSR-VTT (52.8%), MSVD (48.2%), ActivityNet (48.9%), and DiDeMo (50.1%). The code of this paper is available at https://github.com/bzy-source/SMA.