FAESR: Fine-Grained Rate Adaptation for Energy-Aware Super Resolution in Mobile Panoramic Video Streaming

Super resolution (SR) has been proposed to reduce the bandwidth overhead and improve the user’s quality of experience (QoE) for panoramic video. However, video reconstruction greatly increases the energy consumption on mobile devices with limited battery capacity, which is rarely considered in the existing works. In this work, we propose FAESR, a Fine-grained bitrate Adaptation method with an Energy-aware Super-Resolution to maximize the QoE and minimize the energy consumption. We propose an SR power model, which is the first model to evaluate the power consumption of SR on mobile devices through actual measurements. We formulate a joint optimization problem for QoE- and energy-aware panoramic video streaming. Most neural-enhanced panoramic streaming methods use coarse-grained adaptation, either selecting only download bitrate or assigning uniform bitrate to within the predicted field of view (FoV). This can lead to bandwidth waste due to overrated tiles incorrectly predicted to be within the FoV. We develop a fine-grained bitrate adaptation algorithm based on branching sequential DRL, which jointly optimizes download and reconstruction bitrates at the tile level. Evaluation results demonstrated that FAESR can significantly reduce energy consumption by 28% while improving the QoE by 12% compared to the existing state-of-the-art works.