A 46 TOPS/W In-/Near-Memory Computing Processor for Large Language Model with Extended Sparse Attention

A highly energy-efficient embedded DRAM (eDRAM)-based heterogeneous-in/near-memory computing (H-INMC) processor for large language model with extended sparse attention is proposed. The proposed H-INMC processor reduces external memory access (EMA) and enhances macro/system energy efficiency through three key features: 1) attention block fusion computation strategy to maximize input and intermediate data reuse, achieving 85.86% EMA reduction; 2) H-INMC architecture for addressing the imbalances in memory and computation intensity across different computation stages, reducing 77.27% of system latency; and 3) cross-read 3T1C bitcell architecture for mitigating read/write datapath conflictions between CIMs and enhancing the energy efficiency up to 46 TOPS/W with dual-row in-memory-computation. Designed with 28 nm CMOS technology, the proposed H-INMC processor achieves a 92.41% F1 score, evaluated under the Bigbird-large model on the SQuAD 1.1v dataset.