Oaci: Online Adaptive Collaborative Inference Among Edge Devices Under Resource-Constrained Conditions

Under resource-constrained conditions, the performance level of edge devices, including limited device memory and the ability to load models, significantly impacts inference on these devices. Additionally, network fluctuations further exacerbate inference delays during collaborative inference across multiple devices. Therefore, to effectively utilize the resources of edge devices and address the impact of network fluctuations on collaborative inference, this paper proposes an Online Adaptive Collaborative Inference among edge devices under resource-constrained conditions (OACI)method, which splits lightweight ONNX models for deployment across multiple edge devices. Notably, the model's accuracy is not affected after partitioning. Our enhanced <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\lambda$</tex> LinUCB algorithm dynamically predicts inference delays and optimizes partition points for cooperative inference. Experiments on real and simulated data show that OACI achieves 97.71 % accuracy in predicting inference delays and reduces CPU utilization to one-third of that required for single-device inference. This significantly alleviates the inference pressure on edge devices under resource-constrained conditions.