Class-Aware Caching for Convolutional Neural Networks

Many computer vision based mobile applications, ranging from simple children's games to complex, safety-critical applications use convolutional neural networks (CNNs) to provide accurate image recognition of both still images and live video streams. Current state-of-the-art CNNs, however, are very computationally complex and not well suited to real-time mobile computation, where computing resources are severely limited. Current work to improve the latency of CNNs on mobile devices include model compression, caching, as well as CNNs designed specifically for mobile use (e.g. MobileNet). All of these approaches, however, suffer from significantly reduced accuracy. In this paper we introduce a class-aware caching scheme for CNNs that provides fast, high-quality image recognition while not sacrificing accuracy. During offline computation, our system learns the most valuable filters for each class of a dataset through our proposed innovative filter ranking algorithm. At inference time we quickly determine the top-5 most likely classifications of an input image using an LSH, which is very fast but not as accurate as a CNN. Then, using the filter rankings we compute the valuable filters for the top-5 predicted classes while re-using previous computations for any others. This allows the CNN to maintain flexibility, and robustness which are often lost with model compression or mobile-based CNNs, while still providing significant latency reduction. Our evaluation shows that we can achieve latency reduction of up to 30 % while providing a similar level of accuracy.