Proposal for a Job Management Tool for Deep Learning in a Mixed Environment of Heterogeneous GPUs

In recent years, the use of GPUs for computation has become commonplace in deep learning. It is not uncommon for deep learning labs to procure GPU servers for computation without standardising specifications such as GPU architecture and VRAM capacity, resulting in a heterogeneous GPU server environment. Container technology is also widely used as a method of isolating software execution environments. Among them, Docker is the de facto standard for container platforms. In this paper, we have developed a management system that can allocate deep learning training jobs to GPU servers (workers) using container technology to absorb differences in execution environments in heterogeneous GPU server environments. In experiments, we confirmed that the proposed system can estimate the completion time of training for each worker and can detect Out-Of-Memory (OOM) of GPUs in advance. We were able to confirm in advance the “inference time” and the “possibility of OOM”, which are difficult to predict when performing deep learning. We have succeeded in obtaining the “predicted value of learning time” and “possibility of OOM” that engineers need to know in advance, and have shown that the proposed system is effective in deep learning tests.