AdaptServe: Auto-Scalable DL Serving with Dynamic Model Parallelism

With the widespread adoption of large-scale deep learning models in cloud computing environments, model serving systems are facing increasingly higher demands in terms of throughput and latency for online inference. Existing model serving systems often face trade-offs between resource utilization and scalability. Offline placement systems struggle to adapt to real-time workload fluctuations, while online systems suffer from limited granularity in scheduling decisions. These limitations can lead to low resource utilization and high request latency. To address these challenges, we propose AdaptServe, a scalable and dynamic deep learning (DL) model serving system that combines parallel placement strategy with real-time model replicas management to optimize system throughput, scalability, and reliability. During the model initialization phase, AdaptServe employs Integer Linear Programming (ILP) to efficiently allocate resources and place parallelized replicas. In runtime replica management, AdaptServe leverages deep reinforcement learning (DRL) to adaptively manage model replicas in response to fluctuating workloads, achieving real-time auto-scaling model inference services. Experiments show that AdaptServe can cope with different cluster sizes, workload pressures and latency constraints, delivering superior performance compared to state-of-the-art systems while utilizing less than 80% of computational resources.