Selective Layer Acceleration with Data-parallel Architecture for DNN Inference Throughput Maximization

Deep neural networks (DNNs) are widely deployed in CPU-GPU heterogeneous computing systems, where GPU accelerators are often blindly preferred over CPUs. However, accelerating all DNN layers can result in imbalanced CPU and GPU utilizations with suboptimal performance. As a solution, this study presents a selective layer acceleration method with a data-parallel architecture to maximize the DNN inference throughput. Our data-parallel architecture utilizes all the CPU cores while accelerating carefully selected DNN layers. We find an optimal trade-off between the acceleration gains and blocking losses by various layer mappings. Also, analytical bounds for the best- and worst-case response times (BCRT and WCRT) are derived and validated. Our stepwise optimization effectively reduces the search space, finding optimal solutions with minimized optimization times. The implementation of the found optimal mapping shows a 25% inference throughput improvement over the GPU-only mapping.