Achieving Trusted GPU Allocation: An Empirical Study on Efficiency Changes of Deep Learning Training Tasks

In recent years, with the emergence of large models that have extremely high parameter counts and require vast amounts of training data, the number of tasks demanding multiple GPUs surges, promoting the development of computing platforms equipped with substantial GPU resources. A trusted GPU allocation scheme is an essential part of building a trusted deep learning computing platform. Quantifying the relationship between the number of GPUs allocated and the efficiency of deep learning training tasks helps the platform’s scheduling system provide users with more convincing GPU resource allocation plans, improving task execution efficiency, ensuring fairness among tasks, and reducing and stabilizing energy consumption. However, this topic does not receive significant attention and research. In this paper, we conduct an empirical study by deploying different types of deep learning training tasks on the multi-GPU server to explore the relationship between GPU allocation and task execution efficiency. We summarize the efficiency change patterns of various types of tasks under different GPU allocations. Experiments on various training tasks indicate that models with different parameter sizes exhibit different trends as the number of allocated GPUs increases. This study analyzes these trends and uses function fitting methods to describe the efficiency changes of similar tasks.