Interference-Aware DNN Serving on Heterogeneous Processors in Edge Systems

With growing demands on the acceleration of machine learning (ML) computation, processors for edge systems have been integrating heterogeneous devices such as GPUs and NPUs (Neural Processing Units) for ML computing. However, as multiple ML models need to be processed simultaneously even in edge systems, ML inference schedulers for such heterogeneous devices must not only consider the efficiency of devices for different ML models, but also consider the interference among them carefully. Based on our analysis on the behaviors of inter-device interference, the study first builds an interference prediction scheme using a multilayer perceptron model. The interference prediction model is trained with the data from randomly generated ML models, and thus it can be prepared without any prior knowledge of actual target ML workloads. Using the highly accurate prediction model, this study proposes a goal-independent scheduling framework, which allows any scheduling objective set by users. The scheduling framework uses a sampled simulation method to support such flexible scheduling goals with a minimized latency. Our experimental results on a commercial edge system show that our framework backed by the interference prediction model can effectively improve performance for diverse goals.