Enabling Efficient Synergistic Multi-view Inference Across Heterogeneous Edge Devices

Multi-view inference (MVI), which accepts images from multiple viewpoints as input of deep neural networks, is proposed to improve the inference accuracy of conventional single-view models. However, existing mechanisms face challenges in feature fusion and computation efficiency: (1) features from inter-view and intra-view contribute differently to inference, and uniform feature fusion limits MVI accuracy; (2) the sophisticated process and tremendous computational workload of MVI cause a considerable increase in inference latency. This article addresses the above challenges and enables high-accuracy and low-latency MVI for edge intelligence by proposing an end-to-edge synergistic multi-view inference (SMVI) framework. SMVI integrates the f eature f u sion module based on pairwise m utual- a ttention (FUMA), which incorporates the differences between features, enhancing MVI accuracy. To optimize the computation of FUMA-based SMVI, we present a joint optimization algorithm of r esource a llocation and m odel p artition (RAMP) to reduce MVI latency, considering device heterogeneity, dynamic network connection, and resource limitation in heterogeneous edge environments. We developed an SMVI prototype system with heterogeneous embedded GPUs and evaluated its performance in real-world MVI scenarios. Extensive experiments demonstrate that the proposed mechanism achieves a notable MVI accuracy improvement of approximately 4% and accelerates the process by 4.08 × compared to state-of-the-art approaches.