Elevating Video Retrieval Capabilities: A Cross-Modal Approach Utilizing Text and Image Generative Models

In this study, we propose a new methodology to enhance the accuracy of video retrieval by integrating rapidly developing text and image generation technologies. The core of our study is to transform search queries into a more diverse range of search requirements using ChatGPT, and convert them into more specific visual features using stable diffusion, thereby accurately retrieving video content that is highly relevant to search queries. In our experiments, we used a large-scale video dataset from the TRECVID benchmark, which is considered the industry standard as concerns diversity and inclusiveness, to validate the effectiveness of our method. The combination of the advanced natural language understanding capabilities of ChatGPT and image generation prowess of stable diffusion enabled us to successfully retrieve highly relevant search results, even for complicated queries.