AdapCP: Collaborative Inference with Adaptive CNN Partition on Distributed Edge Servers

Due to the limited resources of end devices, the task of Convolutional Neural Network (CNN) inference on the end-side is moving towards edge-end collaboration. However, existing collaborative methods mainly focus on offloading CNN inference tasks from end devices to a single-edge server, which leads to inefficient use of computational resources among nearby edge servers. Moreover, offloading the CNN inference task to a single third-party server may raise privacy concerns. To address these challenges, we propose a framework named AdapCP that introduces a collaborative and adaptive parallel acceleration strategy that utilizes the end device and multiple edge servers. AdapCP consists of two stages: (1) offloading to nearby servers and (2) parallel processing of the CNN inference. For the offloading phase, we use integer linear programming to find the partition points at the inter-layer level. For the parallel phase, we first investigate intra-layer structural splitting methods tailored for both convolutional and fully connected layers. Then, we employ a Deep Deterministic Policy Gradient (DDPG) algorithm based on the Dirichlet distribution to decide the partition points. Finally, we set a periodic update index to enhance AdapCP’s adaptability to dynamic environments. Empirical evaluations conducted on the Jetson nano demonstrate that AdapCP significantly reduces the total latency of CNN inference by an average factor of 2.21 \(\times\) compared to existing solutions.