A 3.51 TOPS/mm ² Transformer Accelerator Exploiting Bipolar Sparsity and Approximate Gating

Transformer models excel at natural language processing tasks but are challenging to deploy on edge devices due to high memory and computation demands. To address this, we proposed an energy- and area-efficient transformer accelerator. We identify ‘0’ bits in positive and ‘1’ bits in negative 2’s complement activation values as bipolar sparsity. This form of sparsity shares a larger proportion than the traditional bit-level sparsity in transformer models. We propose a bipolar sparsity compressor (BSC) together with a bipolar processing element (BPE) to detect and skip the bipolar sparsity in a bit-group (BG) level during the inference. It reduces a large proportion of ineffective computations and improves throughput. The significant sparsity scheduling (SSS) dynamically adjusts broadcast settings based on BG-level sparsity ratios, balancing the sparsity skipping and memory access. Furthermore, an importance approximate gating (IAG) filters out unimportant tokens/heads during the attention computation by reusing sparsity information from the BSC, further reducing processing latency and energy consumption. Implemented in a 28nm process, the proposed accelerator achieves <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$7.62\times $</tex-math> </inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$12.43\times $</tex-math> </inline-formula> throughput improvements on RoBERTa-B and GPT2-xl, respectively. The area efficiency reaches up to 3.51 TOPS/mm<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> due to skipping a large proportion of bipolar sparsity, reaching <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$4.83\times$</tex-math> </inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$5.85\times $</tex-math> </inline-formula> higher compared with the state-of-the-art approximate computing accelerator and the computing in memory accelerator on the benchmark model.