CELLA: A 28nm Compute-Memory Co-Optimized Real-Time Digital CIM-Based Edge LLM Accelerator with 1.78ms-Response in Prefill and 31.32 Token/s in Decoding

This work presents a digital Compute-in-memory (CIM)-based Edge LLM Accelerator CELLA with Compute-Memory co-optimization for Prefill-Decoding phases: 1) Prefill is compute-bound and CIM works in MAC-mode. A puzzle-shaped input scheduling unit (PISU) reassembles input into an interlocking pattern, reducing <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.82\times$</tex> TTFT. 2) Decoding is memory-bound and CIM works in CAM-mode. (a) For static parameters, non-uniform quantization multiply-accumulation (NUQ-MAC) is processed in CIM with majority-first index searcher (MFIS), saving <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$4.89\times$</tex> TPOT. (b) For dynamic KV cache, only important KV is fetched by hash-based cache filter (HCF), saving <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$4.13\times$</tex> TPOT. It reaches 1.78ms-response and 31.32token/s, speeding up <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$2.53\sim 9.74\times$</tex> than state-of-the-art accelerators and edge GPU.