PASS: A Priority-based Model Assignment for Minimal Inference Time in Serverless Edge Cloud

Serverless computing is increasingly being adopted to provision various on-demand services at the edge cloud, including inference tasks based on deep neural networks (DNNs) for the Internet of Things (IoT). This approach leverages the advantages of flexible resource allocation and fine-grained resource management. However, the provisioning of on-demand inference typically requires downloading the DNN model at runtime, which can introduce significant delays. In the edge cloud with heterogeneous network connections, the inevitable model downloading time and the inter-model data transmission impose high challenges to the QoS of inference tasks. In this paper, we investigate how to jointly consider both model downloading time and communication time to minimize inference time. We first formulate this problem into a nonlinear optimization form and proved it as NP-hard. We further propose a Priority-based Model Assignment (PASS) algorithm in polynomial time and trace-driven experimental results show that it reduces the average inference time by 23.6% compared to existing state-of-the-art solutions.