Computational resources for LLM computing: The need for GPU virtualization

With the rapid advancement of AI technologies like Large Language Models (LLMs), both training and deploying these models have become increasingly resource-intensive, particularly due to the heavy demands on hardware such as General-Purpose Graphics Processing Units (GPGPUs). Additionally, relying heavily on cloud infrastructures can lead to high operational costs and underutilized GPU resources. To tackle these challenges, recent research has investigated GPU virtualization as a promising approach to enhance flexibility and optimize resource utilization. In this paper, we begin by examining the growing demand for resources in LLM training and inference workloads, followed by an analysis of various virtualization techniques, highlighting their benefits and limitations. Next, we describe the details of our open- source GPU virtualization framework GVirtuS, its components and the way how it deals with executing CUDA functions on behalf of devices or VMs without direct access to a GPU. Our results demonstrate that GVirtuS can delivers up to a 2× speed-up for LLM inference compared to CPU-only processing. As part of our EU Chips-JU project CLEVER, this work aims to explore the potential of our GVirtuS framework for broader applications in GPU virtualization and AI workloads in the future.