LRSR: Hardware-Friendly Approximation of Per-Element Quantization via Low-Rank Scale Reconstruction

Post-training quantization is a crucial component for deploying Large Language Models (LLMs) in resource-constrained environments. While fine-grained methods (e.g., per-group) offer high accuracy, they heavily rely on hardware-specific implementations and suffer from memory-bound computations, limiting their applicability on standard matrix multiplication accelerators. Conversely, hardware-friendly quantizations (e.g., per-channel) experience severe accuracy degradation at low bit-widths. To address these limitations, we propose Low-Rank Scale Reconstruction (LRSR), a novel quantization method that approximates per-element scale matrices using an upper-bounding rank-1 decomposition, enabling mathematically correct reconstruction via general matrix multiplication (GEMM). Furthermore, we introduce several synergetic clusterization techniques to significantly boost accuracy and allow optimal hardware load balancing. Experimental evaluations on Qwen3 models demonstrate that at INT4, LRSR reduces KL-Divergence by up to 31% compared to the per-channel baseline, while utilizing 29x less metadata memory compared to per-group (g=128) quantization. This establishes a new Pareto frontier for near-zero overhead, hardware-friendly and calibration-free post-training quantizations.