Pod Placement Techniques to Avoid Job Failures Due to Low GPU Memory in a Kubernetes Environment with Shared GPUs

In a container-based cloud environment, GPUs have the advantage of providing high-performance computation to multiple users, and through GPU sharing, many GPU container users can be accommodated over the number of physical GPUs. This increases resource utilization and minimizes idle time. However, extended resources used to share GPUs in Kubernetes do not partition GPU resources or limit usage and only logically increase the number of GPUs that can be recognized. Therefore, usage limits and equal use of GPU resources cannot be guaranteed among pods sharing GPUs. Additionally, GPU memory generally does not allow for overuse. As a result, if a pod with high GPU memory usage runs a GPU task, it will be unable to limit GPU memory usage and free up available GPU memory. Data must be loaded into the GPU memory to perform the GPU task. However, if the data to be used for computation cannot be loaded into the GPU memory due to insufficient GPU memory, the pod will not be able to start the task and will fail to execute. This paper proposes a pod placement technique to avoid GPU memory shortage when sharing GPUs between pods in the Kubernetes environment. The proposed technique monitors the GPU memory usage and usage frequency of each worker node that makes up the cluster and places the pod on the worker node with the most available GPU memory based on the monitoring information.