Improving the efficiency of streaming by super-resolution-aware encoding of the content

In recent years, we have seen significant progress in advanced image and video upscaling techniques, sometimes called super-resolution or AI-based upscaling. Such algorithms are now broadly available in the forms of software SDKs, as well as functions supported by GPUs on many devices. However, to effectively use such technologies in streaming applications, one must also understand their impacts on visual quality and develop a method that enables using such technologies with maximum effect. This paper brings several considerations towards achieving such a goal. We first present a brief survey of the available SDKs and libraries for adding super-resolution functionality to streaming players. Next, we propose a parametric quality model for modeling the effects of different upscaling techniques. We validate it by using an existing widely used dataset with subjective scores. Finally, we present a method of generating Super-Resolution-Aware (SRA) encoding profiles for ABR streaming applications, maximizing the average quality of video the streaming system delivers to a population of clients with a known distribution of display sizes and network bandwidth statistics.