FMC-LLM: Enabling FPGAs for Efficient Batched Decoding of 70B+ LLMs with a Memory-Centric Streaming Architecture
For large language model (LLM) acceleration, FPGAs face two challenges: insufficient peak computing performance and unacceptable accuracy loss of model compression. This paper proposes FMC-LLM to enable FPGAs for efficient batched decoding of 70B+ LLMs.
Authors
- Shiyao Li (ORCID: https://orcid.org/0009-0007-4755-6881)
- Guohao Dai (ORCID: https://orcid.org/0000-0003-0849-3252)
- Xinhao Yang (ORCID: https://orcid.org/0009-0001-9739-2930)
- Hongyi Wang (ORCID: https://orcid.org/0009-0008-7095-7963)
- Shulin Zeng (ORCID: https://orcid.org/0000-0002-1030-3748)
- Ziming Zhang (ORCID: https://orcid.org/0009-0009-2632-4824)
- J.J. Wang (ORCID: https://orcid.org/0009-0005-4899-1071)
- Jintao Li (ORCID: https://orcid.org/0009-0006-8903-8276)
- Tsung-Yi Ho (ORCID: https://orcid.org/0000-0001-7348-5625)
- Wenheng Ma (ORCID: https://orcid.org/0000-0003-2349-7286)
- Libo Shen (ORCID: https://orcid.org/0009-0000-1717-2093)
- Tengxuan Liu (ORCID: https://orcid.org/0009-0007-8943-1577)
- Xuefei Ning (ORCID: https://orcid.org/0000-0003-2209-8312)
- Yuhan Zhang (ORCID: https://orcid.org/0000-0001-7497-7998)
- Yu Wang (ORCID: https://orcid.org/0000-0001-6108-5157)
- Hongfu Guo
- Zhenhua Zhu (ORCID: https://orcid.org/0009-0007-9259-7180)
Institutions
- Chinese University of Hong Kong (HK)
- Shanghai Jiao Tong University (CN)
- Tsinghua University (CN)
Publication Details
- Published
- 2025-02-26
- DOI
- https://doi.org/10.1145/3706628.3708863
- Citations
- 2
- Primary Topic
- Algorithms and Data Compression
- Type
- article
- Field-Weighted Citation Impact
- 4.96