Hardware-Efficient Activation Circuit for Edge Computing: Shifter-Polynomial Approximation of Leaky ReLU

Artificial intelligence (AI) processors offer significant progress for edge computing devices in the Industrial Internet of Things (IIoT) domain, promoting a revolution in computational intelligence and efficiency. A hardware-efficient leaky rectified linear unit (ReLU) activation function with polynomial approximation and shifter implementation is proposed to facilitate the deployment of AI processors in edge devices. The constant parameters of the leaky ReLU are approximated using second-order or higher-order polynomials and transformed into arithmetic operations that can be achieved by a combination of shift, addition, or subtraction operations. Furthermore, the approximate circuit modules of the leaky ReLU are developed by employing shifters, adders, and subtractors instead of multipliers and dividers. The experimental results prove that the cubic or higher-order polynomial approximation of leaky ReLU has a negligible impact on the accuracy of cutting-edge deep neural networks. Moreover, approximate leaky ReLU modules with low power consumption, low latency, and a small circuit footprint are demonstrated by substituting a multiplier or divider with a combination of shifters and adders or subtractors. The proposed approximate leaky ReLU techniques have the potential to produce energy-efficient, low-latency, and compact circuit solutions for AI processors in IIoT.