A Reconfigurable Digital Compute-in-Memory Architecture With 2:4 Compressed Weight Storage for Highly Sparse Fusion Neural Network

The fusion of convolutional neural network (CNN) and spiking neural network (SNN) harnesses their complementary strengths in edge video tasks. However, ongoing challenges remain in the low compatibility between two neural networks and the shortage of storage resources. In this brief, we propose a reconfigurable digital compute-in-memory (RDCIM) architecture featuring 2:4 structured weight pruning, tailored for the efficient fusion of CNN and SNN. Specifically, differential frame convolution (DFC) is adopted in place of conventional convolution. Therefore, the RDCIM fusion accelerator can operate in either DFC mode or SNN mode based on the sparsity of activations. First of all, fine-grained 2:4 structured sparsity enables compressed weight storage, thereby doubling the throughput. Secondly, a novel bitwise in-memory Booth multiplier is implemented to support both 4-bit and 8-bit activations, offering flexible computation. Thirdly, the high sparsity inherent in both differential frames and input spikes allows for the concurrent support of three sparse optimization strategies: coarse-grained zero skipping, medium-grained bank gating, and fine-grained precharge gating. Implemented by TSMC 28nm technology, our proposed architecture achieves the peak energy efficiency of 42.02 TOPS/W for DFC and 0.02 pJ/SOP for SNN.