Multi-Grained Alignment with Knowledge Distillation for Partially Relevant Video Retrieval

Partially Relevant Video Retrieval (PRVR) aims to accurately retrieve the most relevant video in response to a query from untrimmed videos. The analysis of video content can be done at three different granularities: frame-level, clip-level, and video-level. Previous methods have focused on one or two of these levels for alignment, limiting the exploration of the video semantics. Moreover, some methods use video-level alignment and apply a self-attention mechanism to generate video-level features, but this may not be ideal as the entire video may not be relevant to the query. We propose a M ulti- G rained A lignment framework with K nowledge D istillation (MGAKD), which purifies the cross-modal alignment knowledge from the Contrastive Language-Image Pre-training (CLIP) model and achieves multi-grained alignment. It extracts cross-modal alignment knowledge from CLIP and imparts this knowledge to the designed student model. For the student model, two branches are designed: an inheritance branch and an exploration branch. The inheritance branch absorbs the knowledge of cross-modal alignment from the CLIP. The exploration branch explores visual features at three granularities: frame-level, clip-level, and video-level. Specifically, we directly align the extracted frame features of the video with the query features to achieve frame-level alignment. In clip-level alignment, the use of Gaussian masks allows for the representation of the beginning, climax, and end of an event. By employing Gaussian masks, we are able to implicitly model clip-level features, resulting in clip features that contain a richer set of contextual information. To further enhance video-level feature exploration, we apply clip-guided attention to generate diverse video-level features based on different queries. This strategy effectively prevents irrelevant video moments from affecting the alignment of videos and queries. We conduct extensive experiments on two publicly available datasets, and the experimental results have surpassed those of the state-of-the-art method, showcasing the superior performance of the proposed method.