An FPGA-Oriented Quantization Approach for Vision Transformer with LUT-Friendly Operations

Vision Transformers (ViTs) have demonstrated remarkable performance in various computer vision tasks. Due to their large model size and computational complexity, model compression is a critical step for deploying vision transformer models on edge devices. This paper focuses on weight quantization, a vision transformer model compression technique that complements weight pruning. Unlike existing quantization methods, we propose an improved quantization scheme that achieves comparable performance with lower bit-width. To better utilize hardware resources, we also introduce an FPGA-oriented quantization approach, which enhances the efficient use of look-up table (LUT) resources in FPGA. Combining these two schemes allows us to maintain accuracy while improving resource utilization and performance of the model on FPGAs. On the ImageNet dataset, our non-zero fix-length quantization scheme achieves 80.74% accuracy at 2 bit-widths and improves accuracy by 0.1% to 0.5% at the same bit-width compared to state-of-the-art vision transformer quantization methods, and scale power-of-2 quantization scheme can effectively utilize LUT resources in FPGA while maintaining an acceptable level of accuracy loss.