A Mixed-Mode Acceleration via Sparsity-Adjustable Pruning for Balancing Computation Density in Lightweight CNNs

Convolutional neural network (CNN) pruning is an effective way to reduce the computation requirement and improve the inference performance of standard convolutional layers. However, as for the low-computation-density layers in lightweight CNNs, pruning not only fails to improve their processing efficiency but also exacerbates the underutilization problem when deploying them on the convolutional engine. To efficiently execute these pruning-ineffective layers and further accelerate lightweight CNNs, a sparsity-adjustable CNN pruning method, which allows the pruning ratio to be adjusted, is proposed to prune the nonpruning-ineffective layers while shielding the pruning-ineffective layers. As a result, it achieves an additional 40% pruning ratio for nonpruning-ineffective layers with only 0.09% accuracy loss. Furthermore, a dense/sparse mixed-mode convolution computation scheme is designed to efficiently process the pruning- and nonpruning-ineffective layers using multiple acceleration techniques. Finally, a lightweight CNN accelerator is implemented on the Xilinx VCU118 FPGA platform. The comparison results with current studies present that this work can realize 1004.2 performance and 0.98 DSP efficiency while deploying MobileNetV2, achieving <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.26\times $</tex-math> </inline-formula>-<inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$6.13\times $</tex-math> </inline-formula> enhancement on DSP efficiency.