An Analog and Digital Hybrid Attention Accelerator for Transformers with Charge-based In-memory Computing

The attention mechanism is a key computing kernel of Transformers, calculating pairwise correlations across the entire input sequence. The computing complexity and frequent memory access in computing self-attention put a huge burden on the system especially when the sequence length increases. This paper presents an analog and digital hybrid processor to accelerate the attention mechanism for transformers in 65 nm CMOS technology. We propose an analog computing-in-memory (CIM) core, which prunes 75% of low-score tokens on average during runtime at ultra-low power and delay. Additionally, a digital processor performs precise computations only for 25% unpruned tokens selected by the analog CIM core, preventing accuracy degradation. Measured results show peak energy efficiency of 14.8 and 1.65 TOPS/W, and peak area efficiency of 976.6 and 79.4GOPS mm<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> in the analog core and the system-on-chip (SoC), respectively.