Optimizing Sparse/Dense VEGETA Accelerator Performance with Microscaling Quantization

Recent advancements in artificial intelligence have led to the development of exponentially larger models, which in turn require substantial amounts of memory. This escalating demand presents a challenge to scaling large models efficiently. One potential solution is quantization, which leverages low bit-width precision to mitigate memory issues. In this study, we explore the integration of low bit-width Microscaling (MX) based quantization into the VEGETA systolic array design. To achieve this, we introduce dedicated buffers within the systolic array architecture to multicast scale factors for both inputs and weights across tiles. Additionally, we modify the adders and multipliers to function using Microscaled floating point (MXFP) formats. SCALE-Sim is adapted to verify our approach, and results indicate that the MXFP format effectively addresses the memory-wall problem, providing a 1.3× speed-up for weight stationary dataflow. In terms of hardware, the MXFP6 (E3M2) implementation delivers significant benefits, including a 4.65× reduction in area, 2.45× lower power consumption, and a 53% increase in maximum operating frequency. Moreover, it achieves an impressive energy efficiency of 7.912 TOPS/W. In conclusion, Microscaling offers a viable and efficient solution to overcome the memory scaling challenges inherent to large AI models.