FSSA: A Fully Serial and Mixed-Precision Systolic Array for Vision Transformer Acceleration on Edge

Transformers have achieved remarkable success, often surpassing human performance in specific tasks. However, this noteworthy capability comes at the cost of increasing complexity, posing significant challenges when deploying on resource-constrained edge devices. Concerning this, previous efforts have primarily focused on enhancing computational efficiency via compression. Leading-edge studies often utilize approximate computing to improve efficiency with minimal loss of accuracy. Meanwhile, mixed-precision accelerators enhance efficiency while preserving accuracy, albeit with extra overhead. Conversely, bit-serial processing presents a compelling alternative to improve efficiency without sacrificing accuracy or increasing overhead. This article introduces the Fully Serial Systolic Array (FSSA) for Transformer acceleration on edge devices. The proposed architecture integrates bit-serial processing into the systolic array architecture to optimize computational efficiency. FSSA leverages a streamlined design, bit-precision adjustment, and bit-sparsity exploitation to accelerate and enhance efficiency. The proposed architecture is described at the RTL level and synthesized using the 28nm technology library, enabling comprehensive evaluation. Furthermore, the functionality of the FSSA was validated through an FPGA-based implementation. Cycle-accurate simulations of Transformers for image classification reveal that a 16×128 FSSA achieves impressive efficiency gains of 68% and 83% in terms of PDP and EDP, respectively. Additionally, FSSA provides further speed-ups of 31% through bit-precision adjustment and 35% through bit-sparsity exploitation for the same workloads without decreasing accuracy. These results highlight the significance of FSSA, outperforming the bit-parallel baseline in computational efficiency without compromising accuracy.