iMAC: Toward Accelerating Deep Neural Networks by Predicting and Removing Ineffectual MAC Operations

Convolutional Neural Networks (CNNs) are employed in a broad range of classification tasks. Despite CNNs have reached high accuracy, they have high computational complexity, so employing this method in many applications such as internet of things (IoT) is challenging. To overcome this challenge, CNN hardware accelerators are proposed. As the main activation function employed in the CNNs is ReLU, a large portion of the output becomes zero. Therefore, sparse CNN accelerators are proposed to reduce the computational complexity. Recent studies show that, rather than employing sparse CNN accelerators, it would be ideal to predict the zero outputs and remove the corresponding computations. In this paper, we propose the iMAC accelerator to bypass ineffectual computations (a computation leads to zero). For this purpose, we propose a prediction unit that determines ineffectual computations without accuracy loss. Then we propose a dataflow to bypass ineffectual computations. The results show that by increasing 12% of area and 18% power overhead, iMAC achieves 2.31× speedup and 1.95× energy efficiency without any accuracy loss. With 3% accuracy loss, it further improves the speed and the energy by 2.94× and 2.49×, respectively. Moreover, compared to the previous prediction method, our method achieves higher speedup and improvement.