RePC: A Novel Neural Video Quality Enhancement System Framework for ABR Streaming of VBR-encoded Videos

With the emergence of next-generation video applications and increasing spatial resolutions, delivering high-quality video is still limited by network bandwidth. Adaptive bitrate (ABR) can select the appropriate bitrate for video streaming based on bandwidth, which can mitigate rebuffering caused by insufficient bandwidth. In comparison to Constant Bitrate (CBR), Variable Bitrate’s (VBR) encoding scheme can achieve the same quality with less bandwidth consumption and is gradually being widely used in ABR streaming. However, the quality of the video is still degraded due to a poor network. Recent research utilizes Super-resolution (SR) in ABR streaming to construct neural Video Quality Enhancement (VQE) systems, thereby improving the quality of video segments downloaded due to insufficient bandwidth. However, SR cannot participate in the downsampling encoding process of videos, which results in the effectiveness of existing SR-based VQE systems being inherently limited due to unavoidable information loss during downsampling encoding. Concurrently, SR’s high computational cost restricts neural VQE systems’ deployment on clients without GPUs. In contrast to the unidirectional workflow of SR, Rescaling can be integrated into the downsampling encoding process of videos, allowing favorable information to be retained for VQE. To implement high-quality real-time VQE for ABR streaming of VBR-encoded videos on CPUs, we propose RePC, a novel neural VQE system framework for optimizing existing neural VQE systems based on Rescaling (Re) for the first time, and Patch Content-awareness (PC). In detail, RePC uses Rescaling instead of SR to achieve better VQE by participating in the video downsampling. We also propose a Video Single-Image Rescaling model, VSIR, to indicate the effectiveness of RePC in quality enhancement. To speed up VQE, RePC designs a PC algorithm to mix interpolation and neural computation based on the practical upsampling ability. Our evaluation results demonstrate quality gains of 0.55–2.96 dB in PSNR and 1.79–3.18 in VMAF with fewer parameters, a speed-up of 15×–286× well up to real-time requirements on CPUs, and Quality of Experience (QoE) improvements of 16.58–26.65 are also achieved in an ABR system under various networking conditions.