Learning hierarchical video-text relationship via large language model for cross-modal video retrieval

With the rapid development of recording and storage hardware, efficient methods to retrieve the desired videos are required. Among the video retrieval methods, cross-modal video retrieval that aims at retrieving a target video from natural language queries has attracted attention. Cross-modal video retrieval is realized by learning a common representation of videos and texts so that their similarity can be calculated directly only based on their contents. However, traditional cross-modal video retrieval approaches only focus on the global features and ignore the fine-grained information such as a single action or event in the video. In this paper, we propose to use the large language model to extract rich action and event information from the text and match them with the paired video hierarchically. We design a prompt to get the semantically informative action and event components in the form of natural language. Experimental results demonstrate the effectiveness of our method.