A One-Shot Floating-Point Compute-in-Memory Macro Featuring PVT Robustness and Mismatch Tolerance for Edge LLMs

In recent years, the emergence of large language models (LLMs) has significantly impacted various industries, enhancing their productivity. The shift towards real-time experiences and privacy concerns has prompted the transition of LLMs from the cloud to edge devices. However, unlike traditional CNNs or transformers, the excessively large size of LLMs presents new challenges for edge devices regarding storage capacity and power consumption. For example, GPT-4 has 1,800B parameters, calling for the low-precision quantization. FP4 demonstrates superior accuracy over INT8 on LLMs [1] with half the storage requirements, as illustrated in Fig. 1. However, prior FP-CIMs [2], [3] are designed for high-precision formats such as FP16/BF16/FP32, posing challenges for an energy-efficient FP4-CIM tailored for edge-LLMs: (1) limited power efficiency and throughput due to pre-alignment operation in prior FP-CIMs [2], [3]; (2) polluted calculation accuracy originated from multiple variations in prior analog MACs [4]–[9]; and (3) the need for hybrid-FP4 formats across different layers of LLMs.