LightningSR: An Acceleration Framework for Single Image Super-Resolution System

Single image super-resolution (SISR) system based on convolutional neural networks achieves major progress. Under normal circumstances, the depth of the network feature learning module will be proportional to its performance. However, deep models have significant limitations in terms of inference time and memory consumption. We discovered that each image has its own unique high-performance inference depth. Based on this idea, we propose a new acceleration framework, LightningSR, which predicts the depth of the information network and skips the subsequent inference as much as possible after sufficient feature learning, directly performing upsampling. In addition, we introduce a new loss-cascade loss to optimize the convolutional structure of multi-scale networks layer by layer. Finally, we developed a special training method for the entire framework, including a similarity tolerance strategy for the depth information network to help it find a reasonable depth for feature learning layers required by different inputs, and a multi-stage training to integrate the entire framework. Experiments on multiple datasets show that our framework can accelerate most SISR methods, especially for large input sizes and deep models.