AESHA: Accelerating Eigen-decomposition-based Sparse Transformer with Hybrid RRAM-SRAM Architecture

Compute-in-memory (CIM) architectures based on emerging nonvolatile memories (eNVM) are recognized as promising candidates for the efficient computation of self-attention-based Transformer models, which are bounded by memory-intensive dynamic computations involving large matrices. However, existing CIM-based approaches mostly focused on the acceleration of vector-matrix multiplications (VMM) to obtain Q, K and V matrices for direct or sparse calculations of the vanilla Transformer. By storing large amounts of intermediate results in eNVM, the endurance limit is ignored which contradicts the reality of these technologies. In this work, a hybrid RRAM-SRAM CIM architecture to accelerate Transformers, namely AESHA, is proposed based on the eigen-decomposition of WQWTK and the systolic in-memory attention reconstruction. By utilizing the features of symmetric/skew-symmetric real matrices, AESHA translates dynamic attention computation to RRAM-based sparse feature transformation and SRAM-based systolic attention reconstruction to fully exploit the advantages of both on the architectural level. Experiments on a broad spectrum of benchmarks demonstrate that AESHA delivers superior performance in terms of pipeline optimization and energy efficiency. Specifically, AESHA achieves 2.59×, 2.69×, 2.62×, 2.58×, 3.08× and 7.84× maximum static RRAM memory footprint reduction in BERT-Base, BERT-Large, BigBird, Sanger, ViT-Base and ViT-Large over vanilla computation stack, respectively. It eliminates all runtime write access to the RRAM arrays to circumvent the endurance limit. Additionally, AESHA demonstrates 3170.0×, 95.6×, 4.1×, 19.0×, and 19.2× speedup, and 336.0K×, 12.6K×, 7.3×, 23.1×, and 25.8× energy reduction over CPU, GPU, ReBERT, ReTransformer, and CPSAA, respectively.