An Accurate and Compact Design Integrating Seven Common Nonlinear Functions in Deep Learning

Deep learning models based on Transformer architectures are undergoing significant transformations as large language models (LLMs) become more widely adopted and computationally demanding. All these models rely on various nonlinear activation functions, such as Softmax, LayerNorm, and other nontrivial activation functions, whose hardware implementations typically consume substantial area and power, mainly due to the complexity of underlying operations, like inverse square root, division, and exponential functions. To address these challenges, this paper proposes a compact and highly shared architecture that can efficiently and accurately compute reciprocals and inverse square roots via algorithm/hardware co-design. Besides, we propose a novel two-staged method for exponential function computation. The above techniques lead to an efficient implementation of LayerNorm and Softmax. Furthermore, those techniques also enable accurate implementations of other common activation functions, such as Sigmoid, Swish, GELU, and Tanh, with a minor extra hardware cost. The proposed 7-in-1 integrated design merely takes 5450μm<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> using a TSMC 28nm process. Experimental results also demonstrate that the proposed design achieves much higher Softmax accuracy than existing approaches.