Towards Floating Point-Based Attention-Free LLM: Hybrid PIM with Non-Uniform Data Format and Reduced Multiplications

Attention-free large language models (LLMs), such as Mamba and RWKV, have emerged as promising architectures to address the quadratic attention complexity of Transformer models. The inference bottleneck of these models lies in memory-bound matrix-vector multiplications (MVMs) and element-wise multiplications (EWMs). The emerging RRAM/SRAM-based Processing-In-Memory (PIM) architectures have shown great potential to overcome the memory wall problem. However, constrained by the supported data format and operator type, directly adopting PIM architectures for attention-free models faces three challenges: (1) RRAM-based analog PIM architectures perform integer (INT) MVMs using voltage, current, and conductance in the analog domain, limiting their application to the more accurate floating point (FP) data format; (2) SRAM-based digital PIM architectures require additional decoder circuits to support FP format, and the SRAM capacity cannot satisfy the storage requirement of LLMs; (3) When performing EWMs using PIM architectures, only one row/column or the diagonal memory cells are activated, resulting in severe device under-utilization.