Co-ViSu: Accelerating Video Super-Resolution With Codec Information Reuse

High-resolution (HR) videos have gained popularity with the widespread adoption of high-definition displays. Super-resolution (SR) techniques aim to recover HR frames from low-resolution (LR) frames. While deep neural network (DNN)-based SR methods have outperformed traditional techniques in quality, they face performance challenges. FPGA-based SR accelerators have been developed to optimize the performance and power efficiency. However, most of these accelerators process only uncompressed video frames and perform per-frame DNN inference, overlooking the temporal-spatial information inherent in compressed video bitstreams. We propose a novel compressed video SR workflow that includes a codec information reuse algorithm and a dedicated FPGA accelerator named Co-ViSu. Our approach leverages the observation that non-key frames can be reconstructed using codec information and HR key-frames, significantly reducing DNN computations. The Co-ViSu algorithm employs subpixel interpolation to enhance high-frequency details and an MV-aware method to improve SR reconstruction quality. The Co-ViSu hardware integrates decoder, SR, and encoder engines within a parallel pipeline architecture, utilizing codec information reuse to bypass non-key frame decoding, eliminate complex DNN computations, and accelerate encoding processes. Experimental results demonstrate that Co-ViSu achieves performance improvements ranging from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.6\times $ </tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$9.4\times $ </tex-math></inline-formula> and a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.2\times $ </tex-math></inline-formula> gain in energy efficiency with minimal quality loss compared to traditional flow. Additionally, Co-ViSu offers a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.1\times $ </tex-math></inline-formula> increase in throughput compared to state-of-the-art solutions.