High-Performance Video Retrieval by Combining Contrastive Language Image Pre-Training and Cross-Modal Attention

Video retrieval is a task that locates the video segments that best match the query text. Although joint language-image models have recently achieved great success, training a video retrieval model typically requires a large data set and leads to long training time. This paper aims for a high-performance video retrieval method by combining pre-trained encoders and cross-modal attention. We first leverage the pre-trained CLIP (Contrastive Language-Image Pre-Training) ViT-B/16 encoders to extract the feature vectors for videos and texts. Next, the X-Pool, a recent cross-modal language-video attention model is incorporated into the system, where three layers of X-Pool are used to enhance retrieval accuracy. We also suggest using sparse sampling on video frames to further reduce training time. The effectiveness of the proposed method is evaluated using the MSRVTT and MSVD benchmark datasets. The proposed method achieves the best retrieval performance (recall@k, median rank, and mean rank) among the state-of-the-art methods with comparable computational complexity.