Clustering-Based-Approach for Hardware Implementation of Activation Functions

In this work, K-Means clustering technique is adopted to realize non-linear activation functions in IEEE Standard Single-Precision Floating-Point (SPFP) data format. Four different design arrangements along the number of clusters ($K=4,8,12,16$) are characterized in terms of hardware parameters and error characteristics to realize Hyperbolic Tangent, and Sigmoid functions, and are further benchmarked against state-of-the-art (SOTA) - CORDIC, Simplified Piece-Wise-Quadratic (SPWQ), and Piece-Wise-Linear (PWL) engineered designs. Among the proposed design variants, the most resource-expensive and accurate design variant $(K=16)$ presented hardware benefits of $72.39 \%, 92.83 \%, 85.79 \%$ in terms of critical path delay, power, and silicon footprint, respectively, over the CORDIC SOTA method for Sigmoid implementation. The least hardware benefits of $75.41 \%, 44.85 \%$, and ${6 . 7 8 \%}$ along footprint, delay, and power costs were achieved by the proposed design variant ($K=16$), over the $S P W Q$ method for realizing Hyperbolic-Tangent function. The proposed clustering based approximate approach for realizing hardware form of activation functions was validated for AlexNet and VGG-16 CNN models, showcasing comparable test accuracy over exact method. Overall, the clustering technique is a unique method that simplifies the hardware realization of non-linear activation functions, for further adoption in Convolution Neural Network (CNN) accelerator design.