AliSum: Multimodal Summarization with Multimodal Output Boosted by Multimodal Alignment

The exponentially surged multimedia data in the contemporary era has urged the research on multimodal summarization with multimodal output (MSMO) which usually aims to effectively distill and summarize such diverse contents for obtaining user's more satisfactory multimodal summary with images and texts. However, most MSMO methods primarily rely more on textual captions of images, whereas sufficient high-quality image annotations are not always available. To maximize the utilization of visual contents and alleviate laborious image annotation in the summarization process, a novel AliSum is proposed in this paper, which greatly boosts MSMO by multimodal alignment. It mainly includes three parts, the modal-aware encoder, the multimodal alignment transformer and a decoder. The multimodal alignment enhances abstractive text generation quality and facilitates image reference selection based on text-image similarities. Furthermore, we employ knowledge distillation learning to integrate textual and visual modalities into a shared multimodal embedding space. Additionally, a novel contrastive learning approach, called holistic multimodal contrastive learning, pairs texts with a holistic representation of multiple images in a multimodal article which further improves the accuracy of selected images. The ROUGE scores of 56.92 (ROUGE-1), 30.21 (ROUGE-2), and 56.04 (ROUGE-L), along with the image precision (IP) accuracy of 77.19% all have validated the efficacy of the proposed AIiSum.