PoMQ-ViT: Mixed-Precision Quantization Vision Transformer with Pareto Optimization

Vision Transformer (ViT) has established itself as the leading model in image processing, and quantization offers a practical way to deploy ViTs on memory-constrained devices. This technique maps float32 inference to lower bits such as int8. However, existing studies often use uniform bit-width across ViT encoders, leading to sub-optimal allocation and accuracy loss. This study aims to optimize bit-width allocation by leveraging the observed variation in quantization sensitivity among different ViT encoders. We propose PoMQ-ViT, a method that dynamically assigns bit-width to ViT encoders based on their sensitivity. It uses Pareto Optimization to balance accuracy and inference speed. Experiments on the ImageNet Dataset show PoMQ-ViT outperforms uniform bit-width methods across DeiT and Swin models. Within the same computation budget, it also achieves 0.1–1.5 percentage points higher accuracy than other mixed-precision paradigms across different models. This work demonstrates PoMQ-ViT’s effectiveness in ViT quantization, providing a practical solution for resource-constrained deployments.