MVS-SupCon: multimodal video summarization with supervised contrastive learning

Due to the rapid growth of multimedia data, multimodal video summarization has become a critical tool for addressing information overload and improving content understanding. It integrates multi-modal information such as text and images to extract key content and generate compact and representative summaries. However, the already proposed methods exhibit two key limitations. First, they typically process multimodal inputs (text and video) in the encoder but generate only video-based summaries, neglecting the decoder’s potential to leverage textual insights during decoding. Second, their optimizations focus on modality-specific objectives, neglecting cross-modal interactions crucial for holistic representation learning. To address these issues, we propose MVS-SupCon, a novel multimodal joint encoder-decoder architecture designed to generate dual-modal (text and video) summaries. In addition, we introduce cross-modal contrastive learning to enforce semantic consistency between modalities, ensuring that textual and visual representations align cohesively. Experimental results on the ActivityNet Captions and CNN/Daily Mail datasets demonstrate that MVS-SupCon significantly outperforms baseline models in cross-modal fusion, achieving superior alignment between textual and visual summaries while preserving contextual relevance.