A 28nm 76.25TOPS/W RRAM/SRAM-Collaborative CIM Fine-Tuning Accelerator with RRAM-Endurance/Latency-Aware Weight Allocation for CNN and Transformer

Fine-tuning stands out as an emerging trend in recent machine learning systems, offering significant utility across domains from transfer learning, large language models to privacy-sensitive tasks on edge [1]. In the context of fine-tuning, it is an efficient practice to preserve most parameters of the pre-trained feature extraction layers and re-train the weights associated with the fully connected (FC) layers based on a new yet limited dataset. Embedded non-volatile memory (NVM) based CIMs, such as RRAM, and SoCs feature high area and energy efficiency [2]. Also, the multi-chip RRAM accelerator has realized DNN computing without off-chip memory [3]. However, incorporating fine-tuning or training into RRAM-based hardware poses the following challenges (Fig. 1): 1) The frequent weight updating of fine-tuning is limited by NVM endurance, the finite number of times an NVM cell can be written before it wears out; 2) The weight updating process is subject to notable deceleration due to the write and verification latency of RRAM; 3) Higher precision and more storage are desired to represent weight gradients (WG) in finetuning than inference-only accelerators. Previous RRAM-based neural network accelerators that accommodate fine-tuning [5] often include a dedicated SRAM module to replicate higher bit-width weights, resulting in a large on-chip area and power consumption. RRAM modules are not involved in the training or fine-tuning process.