Super Microscaling: Enhancing Precision and Hardware Efficiency in Deep Learning Quantization

The Microscaling (MX) data format, an state-of-the-art quantization technique for deep learning tensor operations, suffers from precision loss at low bit-widths and potential hardware overhead. To overcome these limitations, we propose the Super Microscaling (SMX) data format and its accompanying hardware engine. SMX ensures higher accuracy by optimizing the conversion from Floating Point (FP) to MX, particularly in low-bit width scenarios. We also present spatial-temporal reuse strategy and two-level dequantization hardware architecture, which boosts efficiency in terms of hardware area and energy consumption. Experimental results demonstrate that SMX outperforms MX, achieving average accuracy improvements of 55.86% for BERT, 40.89% for ResNet50, and 37.51% for VGG, while reducing hardware area and energy consumption by 22.74% and 25.46%, respectively.