Text Query-Guided Video Summarization with Self-Attentive Fully Convolutional Network

The importance of text queries in video summarization has grown significantly as they enable the generation of summaries tailored to individual user requirements. The summarization process in both conventional and domain-specific video summarization methods does not take into account the user’s preferences. Given the perspective nature of video summarization, it becomes crucial to incorporate user preferences while generating the video summaries. This research addresses this challenge by proposing a Self-Attentive Fully Convolutional Network (SAFCN), which takes an input text query from the user and generates a video summary pertinent to the query. The proposed network comprises a video representation unit utilizing a pretrained 3D Convolutional Neural Network (CNN), local self-attention, and query-relevant global self-attention units, which compute the importance scores of video shots based on the input textual query. The model incorporates a 2D positional encoding within the self-attention mechanisms and utilizes the Lion optimizer, along with summary length regularization. Extensive experimentation is conducted on the benchmark Query-Focused Video Summarization (QFVS) and QVHighlights dataset. The proposed method achieves a 5% improvement in F1-score for the QFVS dataset and demonstrates competitive performance on the QVHighlights dataset. Both qualitative and quantitative results illustrate that the proposed approach performs better than state-of-the-art techniques.