A 22-nm 109.3-to-249.5-TFLOPS/W Outlier-Aware Floating-Point SRAM Compute-in-Memory Macro for Large Language Models

Large language models (LLMs) have demonstrated exceptional performance in complex artificial intelligence (AI) tasks. However, their rapidly increasing parameter sizes lead to significant communication and computational overhead, posing challenges to the energy efficiency (EEF) and memory footprint of AI processors. Compute-in-memory (CIM) architecture has emerged as a promising solution to alleviate bandwidth constraints and improve EEF. Nonetheless, both integer (INT) and floating-point (FP) CIM implementations struggle with the trade-off between accuracy and memory requirement when applied to LLMs. Outlier-aware quantization (OAQ), which employs low-precision formats for normal values and retains FP formats for high-magnitude outliers, has proven effective in matching the accuracy of full-FP baselines and has become a mainstream approach for efficient LLM deployment. Therefore, this work presents OA-CIM, an SRAM-based digital CIM macro that facilitates element-wise hybrid processing of BF16 outliers and INT4 normal values. The major contributions are: 1) an LUT-based multiply-and-accumulate (MAC) circuit design, which supports efficient FP/INT-compatible (FIC) MAC operations; 2) an <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">xor</small>-sharing non-maximum exponent gating scheme that reduces latency and area by bypassing unnecessary exponent comparisons in FP dataflow; and 3) a sparsity-aware readout circuit with distribution-offset weight encoding (DOWE) to mitigate the power-intensive charging/discharging process on the bitline. A 22-nm 512-kB 8T SRAM OA-CIM prototype is fabricated, which achieves an EEF of 346.6 TOPS/W in INT4 mode and 249.5 TFLOPS/W in outlier mode, representing a <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.7{\times }$</tex-math> </inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$3.1{\times }$</tex-math> </inline-formula> improvement over state-of-the-art mixed-precision CIMs.