Elastically Scheduling Multiple DNN Inference Jobs on an Edge for High Efficiency and Accuracy

An increasing number of applications running on an edge device use Deep Neural Networks (DNNs) to process sensor data and return inference results to make certain decisions. Thus, one research problem is: for a group of DNN inference jobs to run on an edge device, how to determine the job configuration (i.e., the DNN structure, the number of cores, the amount of resource in each core) for each job, so that each job's deadline and accuracy requirements are satisfied and the overall average job latency and accuracy performances are maximized? However, there has been no previous research for this problem. Prior approaches only focus on selecting or generating a model structure for a single DNN inference job on an edge device, or on deciding the structure and computation resource allocation of each job in running jobs (instead of fine-grained decisions on the number of cores and the amount of resources in each core). It is a challenge to coordinate the high-level fine-grained resource scheduling and low-level structure selection among the jobs to achieve the goal. To handle this challenge, in this paper, we first conduct analysis experiments on CPU, GPU, and TPU devices, which show the importance of coordinately selecting structures for different jobs to avoid resource contention, enabling jobs to share resources to increase resource utilization, and selecting the best job configuration to minimize inference time. Based on these, we propose a system that Elastically Schedules Multiple DNN Inference jobs on an edge device for high efficiency and accuracy (ESIM). ESIM uses both heuristic and RL algorithms to find the problem solution. Both container-based emulation and real experiments on CPU, GPU and TPU devices show that ESIM achieves improvement of up to 35% in average inference time, up to 14% in average accuracy, up to 33% in timeliness guarantee ratio, and up to 27% in accuracy guarantee ratio compared to the state-of-the-art approaches.