An Area-Efficient Accelerator for Convolution and Vision Transformer

Convolutional Neural Networks (CNNs) excel in local feature extraction but struggle to capture global context due to convolution kernel constraints. Vision Transformers (ViTs), utilizing self-attention mechanisms, address this limitation effectively. This paper combines CNNs and ViTs in a universal accelerator, optimizing Softmax computations within the attention mechanism. The proposed design achieves 238 GOPS for convolution and 220.5 GOPS for attention at 500 MHz, with area efficiencies of 103.5 GOPS/mm<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> and 95.9 GOPS/mm<sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup>, respectively, enabling efficient processing of various vision tasks.