T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge
The deployment of Large Language Models (LLMs) on edge devices is increasingly important to enhance on-device intelligence. Weight quantization is crucial for reducing the memory footprint of LLMs on devices. However, low-bit LLMs necessitate mixed precision matrix multiplication (mpGEMM) of low precision weights and high precision activations during inference. Existing systems, lacking native support for mpGEMM, resort to dequantize weights for high precision computation. Such an indirect way can lead to a significant inference overhead.
Authors
- Lei Wang (ORCID: https://orcid.org/0009-0006-2313-5348)
- Jianyu Wei (ORCID: https://orcid.org/0009-0009-0830-044X)
- Shijie Cao (ORCID: https://orcid.org/0009-0000-2001-3763)
- Mao Yang (ORCID: https://orcid.org/0009-0009-6455-3898)
- Ting Cao (ORCID: https://orcid.org/0000-0002-9107-013X)
- Lingxiao Ma (ORCID: https://orcid.org/0009-0009-9524-5476)
- Yanyong Zhang (ORCID: https://orcid.org/0000-0001-9046-798X)
Institutions
- Microsoft Research (India) (IN)
- Microsoft Research (United Kingdom) (GB)
- University of Science and Technology Chittagong (BD)
Publication Details
- Published
- 2025-03-26
- DOI
- https://doi.org/10.1145/3689031.3696099
- Citations
- 10
- Primary Topic
- Advanced Data Storage Technologies
- Type
- article
- Field-Weighted Citation Impact
- 14.87