Towards Efficient Sparse Deep Neural Network Inference via Multi-level Concurrency Orchestration

Recent years have witnessed the prosperity of Sparse Deep Neural Networks (SpDNNs), as they are more parameter-efficient and cost-effective than their dense counterparts. To further reduce costs, it is prevalent for AI companies and research groups to manage GPU servers in a multi-tenant manner that handles multiple inference requests simultaneously. However, SpDNNs suffer from intricate data dependency and low GPU resource utilization, which impedes the efficiency of batched inference. To tackle the above problems, we propose CoSpDNN, which optimizes SpDNNs through exploiting intra-kernel and inter-kernel concurrency on GPUs. On one hand, we design a fine-grained pipeline in inference kernels to minimize bubbles and improve concurrency among GPU resources. On the other hand, we develop an adaptive scheduling strategy to enable efficient spatial sharing of GPUs among multiple inference tasks. We evaluate CoSpDNN through extensive experiments, and the results demonstrate that our work outperforms previous champions in task completion time on a single SpDNN and diverse SpDNNs.