Gop-Level Adaptive Resampling with CNN-based Super Resolution

Recently, deep learning-based super resolution methods have been studied for resampling-based video coding to compress high resolution images with limited bandwidth. In this paper, a group of picture-level (GOP-level) adaptive resampling method with convolutional neural network-based (CNN-based) super resolution is proposed to improve the coding gains beyond the latest video coding standard, named Versatile Video Coding (VVC). Specifically, to better restore the detailed information of high resolution video, a super resolution network using multiple side information is first proposed for generating the up-sampled videos. Besides, to further improve the overall performance, an encoder decision strategy is proposed to adaptively select the best scale factor from ×1.0 (original size) and ×2.0 (half size) to determine the encoding resolution at the GOP level. Experimental results demonstrate that the proposed method achieves {-5.34%, -2.38%, -2.08%} and {-3.35%, -8.02%, -4.98%} BD-rate savings for {Y, U, V} under random access and all intra configurations, respectively. This proposed method has been adopted into the reference software of JVET-NNVC.