Video Summarization Algorithm Based on Multimodal Feature Fusion

The video summary generation algorithm constructs a concise and complete summary by selecting the most informative parts of the video content, which is conducive to quickly understanding the video content and compressing the storage space. To address the limitations of existing video summarization methods, such as underutilized multi-modal information and inadequate feature fusion capability, this paper proposes a video summarization algorithm that integrates multi-modal feature fusion with multi-scale temporal modeling. Firstly, a two-stage feature fusion module is proposed based on video image, audio and text features, which fully retains the non-redundant information between modalities and improves the feature representation ability of single frame. Secondly, self-attention and spatio-temporal pyramid pooling are used to model the global and local dependence of the fusion features. Finally, the key frames are selected according to the multi-scale context information to form a high-quality video summary. Experimental results show that compared with other video summarization algorithms, the proposed algorithm achieves the best F-Score on both SumMe dataset and TVSum dataset.