Adaptive GPU Resource Allocation in Online Scenarios

With the widespread adoption of GPUs in recommender systems and large-scale deep learning models, increasing model complexity has intensified challenges related to memory constraints and computational efficiency. Traditional static resource allocation strategies—especially those supporting mixed-model deployments on a single GPU—often lead to reduced GPU utilization, lower cache hit rates, and fluctuating inference throughput. To overcome these limitations and the tight supply of heterogeneous GPU cards, this paper proposes a dynamic GPU resource scheduling system based on a feedback-controlled vGASA algorithm. By continuously monitoring model traffic, per-GPU throughput, and load indicators, our system dynamically adjusts resource allocation and reclamation, ensuring that service level agreements (SLAs) are met while maximizing resource utilization. Moreover, the design isolates resources on a one-model-per-GPU basis, thereby eliminating the overhead associated with computational context switching in mixed deployments. The system supports heterogeneous environments with various GPU types (T4, L20, A10, L40) and diverse inference engines (tf-serving, PyTorch, TensorRT). Experimental deployments on more than 100,000 GPUs have demonstrated significant improvements in resource utilization, with cost savings exceeding 500 million RMB. An extension of the scheduling strategy to CPU-based inference clusters has further reduced process switching overhead, achieving additional savings of over 100 million RMB.