MOFNet: Memory Optimization Fusion Network for Query-Focused Video Summarization

Video summarization aims at selecting key frames or shots with important information from a long video to create a shorter video that captures the essence of the original video content. Query-focused video summarization is a key research focus as it incorporates user subjectivity into the summarization process, further enhancing the user experience. Visual features extraction and multi-modal features fusion are two key technologies for query-focused video summarization, directly affecting the accuracy of the summarization results. However, existing methods neither fully consider the redundancy in extracted visual features nor pay sufficient attention to enhancing the expressive power of multi-modal fusion features. To address these problems, in this paper, we propose a novel Memory Optimization Fusion Network (MOFNet), which focuses on reducing the redundancy of visual features at the extraction phase and exploring the temporal correlation information of visual features during the multi-modal feature fusion phase. Specifically, our MOFNet mainly consists of a global memory optimization module (GMOM) and a cross-modal fusion augmented module (CFAM). Firstly, in the GMOM, we employ a multi-hop operation based on the memory network to optimize the extracted visual features to obtain more useful global visual information. Secondly, in the CFAM, we extract the temporal sequence information and fuse the visual features and query concepts, enhancing the expressive ability of multi-modal features further. Finally, we calculate the shot-level importance score by the fused features after augmented processing, so as to select the shots with high scores to form a video summary. Extensive experiments on two datasets prove the effectiveness of our proposed method.