An Integer-only Quantization Framework for Edge Deployment of Large Language Models

The rapid growth in the parameter size of large language models (LLMs) has introduced significant challenges for deployment on edge devices. To address these challenges, this paper focuses on developing a post-training quantization (PTQ) framework tailored for LLM deployment on edge devices. Our framework introduces an enhanced channel smoothing technique based on channel value ranges, combined with channel reordering to mitigate quantization errors associated with large channel value range differences and activation outliers, reducing the need for quantization and dequantization steps during inference, making it more suitable for edge deployment. Our approach achieves full integer quantization for all model operations, reducing model size by 4×. Through extensive experiments on various language tasks using the OPT model, we demonstrate that our framework surpasses state-of-the-art methods under the W4A4 configuration.