Microscaling Floating Point Formats for Large Language Models

Significant challenges arise from the computational and memory demands for training and inference of large language models (LLMs) as they grow in size and complexity. In order to maximize the accuracy and storage effectiveness of numerical representations within LLMs, this thesis investigates the idea of microscaling floating point formats. Such formats are based on the concept of sharing a single floating-point exponent for an entire block of values, represented by low-precision floating point numbers. Microscaling aims to lower memory footprint and compute overhead by modifying the information representation to reduce the block size employing 8-bit floating point formats and maintaining an acceptable dynamic range introducing the single shared exponent for the block. The focus of this is to study the theoretical foundation and prospective applications of microscaling floating point formats, developing a C++ microscaling framework for boosting the efficiency of LLMs. Through a combination of analytical approaches and exploratory experiments, this thesis intends to build the framework for future study in resource-efficient development and deployment of large-scale language models.