A Fine-Grained GPU Resource Scheduling System for DNN Applications

The significance of GPUs in deep learning applications lies in their powerful parallel computing capabilities, high throughput, optimized hardware architectures, and good support for mainstream deep learning frameworks. DNN applications are widely used for a variety of computation-intensive tasks because of their ability to significantly improve throughput when running on GPUs. However, DNN applications often do not fully utilize GPU resources, and due to the different memory and computation requirements of the operators in the workload, one of the resources may be idle while the operation is executing. To improve the utilization of GPU resources, many existing GPU sharing techniques exist, but there are problems of resource competition and interference in fine-grained scheduling. To solve these problems, we propose a scheduling system, FgGR. We first propose a resource estimation method to estimate the resource requirements of each GPU core as well as the model independently without running the model by analyzing the GPU cores at the bottom of the model. We develop different scheduling strategies for different priority applications, and in order to reduce the interference of low-priority kernels on already running kernels, we propose an interference scoring model to schedule low-priority kernels based on the interference scores and resource requirements of the kernel's tasks, and to realize fine-grained sharing of GPUs. Finally, we conduct experiments on DNN workloads to compare with other baseline models.