WhiteDwarf: 12.24 TFLOPS/W 40 nm Versatile Neural Inference Engine for Ultra-Compact Execution of CNNs and MLPs Through Triple Unstructured Sparsity Exploitation and Triple Model Compression

Although over-parameterization grants neural networks high accuracy in various inference tasks, it also curses them with an exploding computational cost. It is known that these giant models can be sparsified without much loss in performance [1], but to achieve high execution efficiency, they should be condensed as much as possible without hampering the ongoing growth of neural model diversity. This paper presents the following holistic ultracompact approach towards this challenge, as illustrated in Fig. 1, using CNNs and MLPs for the demonstration of its effectiveness: triple model compression utilizing (1) network compression using the Folding technique [2]; (2) a method for concurrent training, pruning, and quantization based on the Multicoated Supermask (M-Sup) algorithm [3, 4]; and (3) adaptive precision with INT2-4 weights, FP8 activations, and FP16 normalization [5]. The proposed WhiteDwarf architecture features (4) off-chip access reduction with Huffmanen-coded weights; (5) triple unstructured sparsity exploitation on a 1 K Processing Element (PE) Tensor, exploiting the high sparsity of activations and weights (Fig. 1, right-top) with a non-zero gathering datapath, and the bit-level sparsity of weights with a bit-serial decomposition of multiplication; and (6) broad support of models via a hybrid controller with a custom RISC-V processor. On ImageNet, we compress a ResNet-50 (RN50) to 5.7% of the original memory size while maintaining 74.7% accuracy, and on CIFAR-100, we compress an MLP-Mixer [6] S24 (MS24) to 0.6% of the original without accuracy loss (see Fig. 4).