Efficient Inference of parallel partitioned hybrid-Vision Transformers

Recent advancements have explored parallel partitioning of Transformers and Convolutional Neural Network (CNN) based models across networks of edge devices to accelerate deep neural network (DNN) inference. However, partitioning strategies for hybrid Vision Transformers---models integrating convolutional and attention layers---remain underdeveloped, particularly in scenarios with low communication data rates. This work introduces a novel partitioning scheme tailored for hybrid Vision Transformers, addressing communication latency through efficient compressed communication and model size reduction. The proposed approach incorporates a trainable quantization and JPEG compression pipeline to minimize overhead. We evaluate our scheme on two state-of-the-art architectures, edgeViT and CoatNet. For a communication data rate of 10 MB/s and partitioning across 12 devices, we achieve up to a 1.74× speed-up and a 5.34× model size reduction for edgeViT-XXS. Similarly, on a customized CoatNet-0, our method achieves a 1.40× speed-up and a 2.66× reduction in model size, demonstrating the efficacy of the approach in real-world scenarios.