Bsqat: block-wise shared quantization-aware training for large language models

Quantization-aware training (QAT) has become a crucial technique for compressing large language models (LLMs), enabling low-bit deployment in resource-constrained environments. However, existing QAT methods often treat blocks in isolation, neglecting structural dependencies between adjacent blocks. This can lead to severe accuracy degradation and inefficient training, especially under ultra-low-bit quantization. To address these limitations, we propose Block-wise Shared Quantization-Aware Training (BSQAT), a two-stage framework that combines local and global optimization for efficient and accurate model quantization. In the Local QAT (LQAT) phase, BSQAT merges adjacent Transformer blocks using an overlapping sliding-window strategy and assigns shared quantization parameters to capture inter-block correlations. To preserve fine-grained weight fidelity, a lightweight low-rank scaling module is designed within each merged block. In the Global QAT (GQAT) phase, global scaling parameters are fine-tuned with minimal overhead to enhance overall model robustness. Extensive experiments demonstrate that BSQAT generalizes well across multiple model architectures including LLaMa, OPT, Vicuna, and Mistral, and scales effectively from 6.7B to 123B parameters. In INT2 quantization of the LLaMa-3-70B model, BSQAT improves average zero-shot accuracy from 65.18% to 67.87% compared to EfficientQAT, while reducing memory usage by approximately 7%.