Mediator: Characterizing and Optimizing Multi-DNN Inference for Energy Efficient Edge Intelligence

Recently, the pushes to execute DNN inference at the edge have been increased. In addition, with more demands for complex functionalities, intelligent services exploit multi-DNN workloads. To accelerate the response of a multi-DNN workload, it is necessary for off-the-shelf heterogeneous processing units to be simultaneously exploited. However, executing a multi-DNN workload on edge devices poses a new challenge - which DNN should run where. Deciding such execution scheduling decision becomes more complicated with the runtime variance (such as resource contention and temperature variation) in the SoCs, which eventually affects the performance and energy consumption. In this paper, we propose a reinforcement learning based multi-DNN inference framework for edge devices, Mediator, which finds energy efficient DNN scheduling decisions. In our evaluation on the state-of-the-art edge devices, Mediator saves system-wide energy consumption by 31.8% and 28.5%, on average (up to 56.8% and 64.9%), compared to the baseline multi-DNN execution techniques and prior multi-DNN execution techniques, while satisfying the latency and accuracy constraints.