RACI: A Resource-Aware Cooperative Inference Framework on Heterogeneous Edge Devices

Cooperative inference for deep neural networks (DNNs) across edge devices has received increasing attention, due to the benefits of low latency, low power consumption, and privacy preservation. Cooperative inference partitions a DNN model into multiple segments, which will then be allocated to distributed devices for parallel inference. Nonetheless, prior works fail to comprehensively study the impact of layer configurations, dynamic network bandwidths, and heterogeneous device capabilities on the inference speed, resulting in suboptimal inference performance. In this work, we conduct a comprehensive analysis of these key factors and figure out the limitations of conventional transfer-based and redundant computation-based methods. Based on the analysis, we first propose a latency prediction agent that accounts for the layer configurations, network bandwidths, and device computing capabilities, aiming to quickly evaluate the inference latency. Furthermore, we propose RACI, a resource-aware cooperative DNNs inference framework on heterogeneous edge devices. It co-trains a model agent for model partition and a workload agent for workload allocation to generate co-optimized model partition and workload allocation strategies, leading to high cooperation inference acceleration. Experimental results demonstrate that RACI outperforms the state-of-the-art approaches by 1.1× -5.2× in terms of inference speedup for three representative DNN models.