Efficient inference pipeline serving by bi-dimensional autoscaling

In recent years, there has been a rise in using multiple deep learning (DL) models chained together, producing intelligent pipelines called inference pipelines. Inference pipelines utilize the scalable cloud resources to predict user queries. Furthermore, they are user-facing, and users expect a timely response to their queries, and long delays negatively impact their experience; this timeliness expectation is addressed by the system guaranteeing an end-to-end latency service level objective (SLO). Inference pipelines receive a fluctuating workload over time due to a variable number of users sending their inference queries. As a result, it is critical to dynamically provision the right resources of the distributed cloud infrastructure to meet the SLOs cost-efficiently. Most current approaches apply horizontal scaling, i.e., deploying more instances of the DL model and load-balancing among them, to meet SLOs of inference pipelines under fluctuating workload. Recent approaches have employed vertical scaling to provide a roughly zero-delay scalability compared to horizontal scaling, which has the cold-start issue for the extra instances. However, vertical scaling is limited by the available local resources and supports fewer queries using the same amount of resources as horizontal scaling. In this work, we propose Biscale to address the problem of efficiently retaining the latency SLO of inference pipelines under workload fluctuation by combining vertical and horizontal scaling. In particular, Biscale uses vertical scaling to react quickly to increased workload demand to retain the SLO and switches to horizontal scaling when the workload stabilizes for cost efficiency. Our preliminary result shows that Biscale has up to 11× fewer SLO violations than horizontal scaling alone, and up to 75% fewer CPU cores than vertical scaling alone.