Hardware-Efficient VLSI Implementation of Self-Attention Transformers for Natural Language Processing

This paper presents an optimized VLSI architecture for implementing transformer-based neural networks with emphasis on hardware efficiency and reduced computational complexity. The proposed design features a six layered architecture with a novel 512-dimensional processing unit utilizing an innovative matrix computation approach that achieves significant power savings. Our implementation leverages CORDIC (COordinate Rotation DIgital Computer) architecture to efficiently compute the trigonometric functions required for positional encoding in the transformer model. The design incorporates two specialized CORDIC modules: one configured in rotation mode for sine/cosine calculations, and another in vectoring mode for division operations, both crucial for positional encoding and softmax computations respectively. The architecture is implemented on Vitrex Ultrascale+ VU37P FPGA which demonstrates improved resource utilization through specialized matrix multiplication units while maintaining the model’s core functionality for natural language processing tasks. The implementation achieves a 30% reduction in power consumption and LUT utilization compared to conventional implementations reduced over 48%.