OPASCA: Outer Product-Based Accelerator With Unified Architecture for Sparse Convolution and Attention

Vision transformer (ViT)-based models have achieved state-of-the-art accuracy in many computer vision tasks, but their attention mechanism is more computation and communication intensive than convolutional neural networks (CNNs). To adapt ViT-based models for resource-constrained edge computing platforms, techniques, such as network sparsity and convolution-attention combination, have been proposed to reduce processing costs without compromising accuracy. Therefore, specific hardware designs able to handle sparsity in both convolution and attention operations are required to accelerate the processing speed. In view of this, this work proposes OPASCA, an accelerator featuring a unified hardware architecture that supports irregular activation and weight sparsity in both operations. Specifically, this target is achieved with the following contributions: 1) we employ outer product dataflow to efficiently handle sparse weights and input neurons, supporting both convolution and attention computing with minimal hardware overhead; 2) we design a hierarchical butterfly network to route the output neurons to the accumulation buffers, minimizing the conflicts among outer product results and reducing the hardware overhead of accumulation banks; and 3) we propose a novel encoding scheme that achieves more compact sparse inputs and enhances multiplier utilization. Evaluations on VGG-16, ViT, BoTNet, and Conformer models show that OPASCA outperforms state-of-the-art accelerators by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.60 \times -2.08 \times $ </tex-math></inline-formula> on sparse convolution tasks and by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.18 \times -1.70 \times $ </tex-math></inline-formula> on sparse attention tasks in term of performance. It also reduces DRAM access to 19%–86% and energy consumption to 36%–92% of those of the counterpart designs.