Microscaling Vision Transformers on FPGAs

Vision Transformers (ViTs) leverage the transformer architecture to effectively capture global context, demonstrating strong performance in computer vision tasks. A major challenge in ViT hardware acceleration is that the model family contains complex arithmetic operations that are sensitive to model accuracy, such as the Softmax and LayerNorm operations, which cannot be mapped onto efficient hardware with low precision. Existing methods only exploit parallelism in the matrix multiplication operations of the model on hardware and keep these complex operations on the CPU. This results in sub-optimal performance due to the communication overhead between the CPU and accelerator. In this work, we propose the first ViT accelerator that maps all operations of the ViT models onto FPGAs with the recently proposed Microscaling Integer (MXInt) arithmetic format. We evaluate how different design choices can be made to trade off accuracy, hardware performance, and hardware utilization. Our contributions are twofold. First, we quantize ViTs using the MXInt format, achieving both high area efficiency and accuracy. Second, we propose MXInt-specific approximation methods that map these complex arithmetic operations into custom hardware. Within 1% accuracy loss, our method achieves at least 93× speedup compared to Float16 and at least 1.9× speedup compared to related work.