MapFormer: Attention-based multi-DNN manager for throughout & power co-optimization on embedded devices

In the context of modern services that use multiple Deep Neural Networks (DNNs), managing workloads on embedded devices presents unique challenges. These devices often incorporate diverse architectures, necessitating advanced management solutions to efficiently deploy multi-DNN workloads. Traditionally, the focus has been on improving throughput, while power optimization has received less attention. This paper presents MapFormer, a new manager that uses attention-based mechanisms to enhance both throughput and power efficiency. MapFormer intelligently assigns multi-DNN workloads to different computing components of embedded systems---CPU, GPU, and DLA---and adjusts operational frequencies to optimize power use. Experimental results show that MapFormer significantly improves average throughput under set power budgets by 90.8%, offering a promising approach for managing complex workloads on heterogeneous embedded systems.