Leveraging Image Difficulty for Run-Time Adaptive DNN Inference on Embedded Devices

Deep Neural Networks (DNNs) impose great challenges on resource-constraint embedded devices since they employ billions of computational operations. To satisfy these computational demands such devices utilize hardware accelerators, which can lead to increased power consumption whatsoever. To that end, the compression of DNNs to lower precision has been proposed, in order to achieve savings in energy consumption at the cost of some accuracy loss during inference. However, DNNs do not behave similarly under lower-precision execution and the accuracy degradation can be severe. In this work, we utilize the notion of image difficulty and explore how we can change DNN precision during inference to achieve gains in energy consumption without big drops in accuracy. We evaluate our work on the ImageNet dataset and show how the proposed framework achieves energy savings at run-time.