Optimizing Deep Neural Network Sparse Kernels on Multi-core CPUs

Vector-wise pruning is widely used to reduce the inference cost of deep neural networks (DNNs), but it turns key operators into sparse convolution (SpConv) and sparse matrix-matrix multiplication (SpMM) kernels that are difficult to optimize on multi-core CPUs. Although prior sparse-kernel acceleration has largely centered on GPUs, CPUs remain a first-order inference platform in CPU-only deployments and in heterogeneous serving when GPUs are scarce or memory-constrained. We present TurboSparse, an approach for accelerating vector-wise sparse DNN kernels on modern multi-core CPUs. TurboSparse (1) eliminates im2col by executing SpConv directly over a DNN-oriented sparse format to reduce data movement and improve reuse, (2) introduces row-merging micro-kernels guided by an analytical SIMD-utilization model to increase vector-lane occupancy, and (3) fuses consecutive sparse kernels when their combined working set fits in private caches to reuse intermediates and cut memory traffic. Across four CPUs (x86 and ARM) and representative pruned DNN workloads, TurboSparse outperforms five expert-tuned CPU libraries, achieving up to 4.05 × speedup for consecutive SpConvs and sustaining 15–80% of each processor’s theoretical peak throughput.