Multi-Modal Video Summarization Based on Two-Stage Fusion of Audio, Visual, and Recognized Text Information

Multi-modal video summarization task provides a text summary by combining information from different kinds of inputs. Previous methods usually generate the summary from video and ground truth text inputs, but seldom study how audio, video and ASR text influence the results. In this work, we introduce a novel multi-modal video summarization approach that leverages video and audio inputs, along with speech-recognized text. Our method initiates by conducting speech recognition using whisper ASR model, followed by representation of visual, textual, and audio information using video transformer, BART, and whisper, respectively. Subsequently, two-stage fusion between different modalities is applied, and the fused features are fed into a BART decoder to generate text summaries. Experiments on the how2 dataset show that our method outperforms previous baselines. It is also revealed that incorporating audio information helps generate better summary even if the texts contain certain errors.