SUM-GAN-FGN: Frequency Domain Unsupervised Video Summarization with Gaussian Noise Based on GAN

Video summarization focuses on extracting a concise set of frames from the original video that retains the most information possible. Most of the current mainstream technologies for video summarization start from the perspective of time domain, framing summarization as a recurrent neural network (RNN) sequence prediction challenge. However, time domain modeling is difficult to use the spatial position information and sound information in the video. In this work, we analyze this problem from a frequency domain perspective. By leveraging the fast Fourier transform (FFT), we extract spectral features and then input it into the Generative Adversarial Network (GAN) to reconstruct the video in an unsupervised manner and then generate a summary. Furthermore, the predicted values for frame importance scores are approximately uniformly distributed, leading to the problem of ineffective feature learning. We add Gaussian noise (auxiliary signal) to frame importance scores to improve keyframe classification accuracy. Considering the subjectivity of labeling for video summarization tasks, we output multiple sets of means and variances of frame importance scores, and use a small, linear neural network layer to learn the best-fit set to improve label confidence. Our experiments on two benchmark datasets (SumMe and TVSum) indicated the competitiveness of our approach. Especially on the former dataset, our F-Score score is as high as 54.5%, achieving excellent performance at present.