SIMD-CP: SIMD with Redundant Bits Compression and Mixed-Precision Packing for Quantized DNNs

Deploying deep neural networks (DNNs) on edge devices presents notable challenges, including execution time, power consumption, and memory footprint. To address these limitations, the co-design of software-based model compression techniques and dedicated hardware has become crucial for the efficient deployment of DNNs on edge devices. However, the hardware needs to support various model compression techniques, and specific compression formats introduce limitations to the effective use of the conventional SIMD, such as low-bit-width precision, fine-grained mixed precision, and sparse matrices. To overcome these issues, we propose SIMD-CP, a SIMD architecture featuring tag-based precision detection and redundant bit-width compression, which is represented as compression packing. Specifically, we introduce two novel SIMD instructions: (i) a tagged vector load instruction ( tvl ), which fetches quantized vectors from memory while appending bit-width metadata as tags, and (ii) a packing dot-product instruction ( pdotp ), which detects the precision levels of elements and packs them into suitable multipliers. Experimental evaluations show that our approach achieves a 2.0× MAC/cycle gain on both fine-grained mixed-precision and sparse-matrix formats by a series of instructions, i.e., tvl and pdotp . Furthermore, SIMD-CP obtains a 2.70 ∼ 3.40× GOPs/W and a 2.31 ∼ 2.42× OPs/LUT improvement for mixed-precision convolution, outperforming the cutting-edge mixed-precision SIMD. These diverse model compression supports allow 28.8 ∼ 45.5% latency reduction for DNN applications, including tiny CNN and edge-aware Vision Transformer, with mitigating accuracy degradation within 1.2 ∼ 2.1%. We also provide the scaling of the SIMD-CP architecture, resulting in a 1.8% LUT utilization increase in the small-scale compared with the conventional mixed-precision SIMD.