A High-Speed and Low-Complexity Architecture for Softmax Function in Deep Learning

In recent years, with the rapid development of deep learning technology, the Transformer model shows superior performance and is widely used in many fields such as natural language processing and computer vision. However, the algorithmic model has high demands in terms of accuracy, real-time and resources, which poses challenges to its hardware implementation. This paper proposes a hardware optimization scheme for the Softmax function widely used in Transformer. By analyzing the computational process of the function, we optimize the computational architecture using mathematical transformations and linear fitting, eliminating complex logic units including division and exponential operations, and significantly reducing the resource consumption. The computational architecture reduces resource consumption by 90% with an average error of less than 5%, and specific parameters for optimization are proposed.