1.536TB/s/mm² Bandwidth Scalable Attention Accelerator with 22.5GOPS Throughput High Speed SoftMax for Quantized Transformers in Intel 3

This work presents a novel hardware accelerator utilizing <3µm pitch 3D Cu-Cu Hybrid bonding interconnect (HBI) technology and specifically designed to efficiently execute Multi Head Attention (MHA) of encoder transformer models, particularly focusing on integer BERT. Transformers have become the dominant architecture for natural language processing (NLP) tasks. We present an accelerator addressing the loss due to low precision model by incorporating specialized hardware optimizations to the quantizer and SoftMax engine. Extremely Wide and variable SRAM/logic Bandwidth for GEMM parallelism including dynamic transpose logic and high speed 2-Pass SoftMax in conjunction offers 22.5GOPS. The Intel 3 prototype with a 3D footprint of 1.2mm2 achieves 25668 Attention/s with no accuracy loss while running I-BERT.