Breakthrough Low-Latency, High-Energy-Efficiency LLM Inference Performance Using NorthPole
For a 3-billion-parameter LLM, a research prototype inference appliance with 16 IBM AIU NorthPole processors delivers a massive 28,356 tokens / second of system throughput and sub-1 ms / token (per-user) latency while consuming merely 672 W for 16 NorthPole cards in a compact 2U form factor. With a focus on low latency and high energy efficiency, when NorthPole (in 12 nm) is compared to a suite of GPUs (in 7/5/4 nm) at various power consumptions, at the lowest GPU latency, NorthPole provides 72.7x better energy metric (tokens/second/W) while providing better latency.
Authors
- Filipp Akopyan (ORCID: https://orcid.org/0009-0001-5312-1311)
- Guillaume Garreau (ORCID: https://orcid.org/0000-0002-4115-9232)
- Myron Flickner
- Takanori Ueda (ORCID: https://orcid.org/0000-0003-1637-4640)
- Rathinakumar Appuswamy
- Michael DeBole (ORCID: https://orcid.org/0000-0002-3068-9430)
- Peter J. Carlson
- Brian Taba (ORCID: https://orcid.org/0009-0008-7831-1139)
- W. P. Risk (ORCID: https://orcid.org/0009-0004-3211-0738)
- Megumi Ito (ORCID: https://orcid.org/0009-0002-3627-5308)
- Andrew S. Cassidy (ORCID: https://orcid.org/0000-0001-7305-4198)
- Deepika Bablani (ORCID: https://orcid.org/0000-0002-8987-8201)
- Rameswar Panda (ORCID: https://orcid.org/0000-0003-4359-2475)
- Ignacio Terrizzano (ORCID: https://orcid.org/0000-0001-6179-2601)
- Steven K. Esser (ORCID: https://orcid.org/0009-0008-4089-0577)
- Rafael T. de Sousa (ORCID: https://orcid.org/0000-0003-1101-3029)
- Jeffrey L. McKinstry
- Nathaniel J. McClatchey
- Dharmendra S. Modha (ORCID: https://orcid.org/0009-0005-6714-8127)
- Yutaka Nakamura (ORCID: https://orcid.org/0000-0002-6921-1266)
- Pallab Datta (ORCID: https://orcid.org/0000-0002-0515-9729)
- Carlos Ortega Otero
- Jeffrey A. Kusnitz
- Jun Sawada
- Jennifer L. Klamo (ORCID: https://orcid.org/0009-0001-9042-0476)
- John V. Arthur
- Neil McGlohon (ORCID: https://orcid.org/0000-0001-8304-4911)
- Rajamohan Gandhasri (ORCID: https://orcid.org/0009-0003-5068-1924)
- Jay Sivagnaname
- Alexander Andreopoulos (ORCID: https://orcid.org/0009-0001-2408-5171)
- Matthew Stallone
- David Cox (ORCID: https://orcid.org/0000-0002-2189-9743)
- Arnon Amir (ORCID: https://orcid.org/0009-0000-6814-165X)
- Tapan K. Nayak (ORCID: https://orcid.org/0000-0001-8941-8961)
- Nelson Mimura Gonzalez (ORCID: https://orcid.org/0000-0002-8313-8114)
- Daniel F. Smith
- Kai Schleupen
Publication Details
- Published
- 2024-09-23
- DOI
- https://doi.org/10.1109/hpec62836.2024.10938418
- Citations
- 3
- Primary Topic
- Image and Signal Denoising Methods
- Type
- article
- Field-Weighted Citation Impact
- 0.75