Optimizing GPU Sharing for Container-Based DNN Serving with Multi-Instance GPUs

The trade-off of throughput versus latency is important in serving deep neural networks (DNNs) on GPUs. A hardware feature---Multi-Instance GPU (MIG)---provides an additional dimension to this trade-off. This paper studies the GPU sharing problem for serving DNNs with MIG. We propose a system Jormungandr that allocates MIG-enabled GPUs to optimize a user-define utility function, and places DNN serving containers on heterogeneous MIG instances. Jormungandr introduces a utility-first search that yields efficient solutions in practice. Additionally, it presents a robust transition protocol that guarantees a seamless switch between two GPU-cluster configurations, minimizing throughput fluctuations. We implement Jormungandr on Kubernetes. Our experiments show that Jormungandr provides near-optimal solutions, using < 6% more GPUs than the optimal solutions.