A New 2-bit Model Quantization Method Based on Non-uniform Quantization

Recently, quantization of large language models (LLMs) has emerged as a crucial technique for model compression, enabling deployment on edge computing devices and enhancing their efficiency. Techniques such as LLM.int8 are widely utilized in the fields of model compression and fine-tuning. However, as the number of quantization bits decreases, these conventional methods often result in a significant degradation of model accuracy. In response to this challenge, we introduce a novel non-uniform quantization approach that meticulously accounts for the influence of activation functions, outliers, and sensitive values on accuracy loss. This method effectively mitigates the accuracy degradation during the quantization process. Our experiments, conducted on the Llama-7B model, demonstrate that our technique achieves substantial compression with 2-bit quantization while preserving high model accuracy.