Power-Limited Inference Performance Optimization Using a Software-Assisted Peak Current Regulation Scheme in a 5-nm AI SoC
Discrete AI inference cards, operating under form-factor and system-defined peak power constraints, must serve diverse inference requests with widely varying power consumption. A peak current-limiting scheme is proposed to maximize inference performance across practical use cases. The peak current management block consists of a card-level current sensing circuit with an AI inference-aware feed-forward and feedback control mechanism. The card-level sensing improves performance by eliminating the need for additional margins for power consumed by off-chip components. Compiler-assisted feed-forward control exploits the predictability of AI inferences and proactively manages peak currents without a static reduction in operating frequency. Measurements from an AI system on chip (SoC), fabricated in 5-nm technology, show up to 41% improvement in Bert-Large inference throughput by engaging the peak current control.
Authors
- Hung Tran (ORCID: https://orcid.org/0000-0001-6836-5151)
- Prasanth Chatarasi (ORCID: https://orcid.org/0000-0002-0974-4001)
- Matthew Cohen
- Gustavo Tèllez
- Alberto Mannari
- Swagath Venkataramani (ORCID: https://orcid.org/0000-0002-0470-6364)
- Matthew M. Ziegler (ORCID: https://orcid.org/0000-0002-9259-7304)
- Zhibin Ren (ORCID: https://orcid.org/0000-0002-4001-1455)
- Radhika Jain (ORCID: https://orcid.org/0000-0003-0896-8131)
- Siyu Koswatta
- Sanchari Sen (ORCID: https://orcid.org/0000-0003-0080-2882)
- Marcel Schaal
- Ching Zhou
- T. Rower
- J. A. Silberman
- Nianzheng Cao (ORCID: https://orcid.org/0000-0003-2786-9139)
- Thomas M Gooding
- Vinay Shah
- Saekyu Lee
- Ashish Ranjan (ORCID: https://orcid.org/0000-0003-2434-0475)
- Xin Zhang (ORCID: https://orcid.org/0000-0002-0579-2268)
- Joshua B. Rubin (ORCID: https://orcid.org/0000-0002-7395-1937)
- Monodeep Kar (ORCID: https://orcid.org/0000-0002-9318-6793)
- Sandra Woodward
- Bruce Fleischer (ORCID: https://orcid.org/0000-0002-7033-1564)
- Kyu Hyun Kim (ORCID: https://orcid.org/0000-0002-5624-8465)
- Martin Lutz
- David Satterfield
- Scot Rider
- Michael Guillorn
- Bahman Hekmatshoartabari
- Vidhi Zalani
- Robert M. Senger (ORCID: https://orcid.org/0000-0002-0360-7014)
- Shubham Jain (ORCID: https://orcid.org/0000-0002-2291-7712)
- Wei Wang (ORCID: https://orcid.org/0000-0002-5453-4695)
- Jintao Zhang (ORCID: https://orcid.org/0000-0001-6909-0612)
- Jinwook Jung (ORCID: https://orcid.org/0000-0002-9384-5277)
- V. Srinivasan
- Ankur Agrawal (ORCID: https://orcid.org/0000-0002-4389-5911)
- Philip Jacob
- Arvind Kumar
- Abey K. Mathew
- Leland Chang
- JohnDavid Lancaster
- Pong-Fei Lu
- Indira Nair
Institutions
- IBM (United States) (US)
- IBM Research - Thomas J. Watson Research Center (US)
- IBM Research - Zurich (CH)
Publication Details
- Journal
- IEEE Journal of Solid-State Circuits
- Published
- 2024-10-18
- DOI
- https://doi.org/10.1109/jssc.2024.3472023
- Primary Topic
- Low-power high-performance VLSI design
- Type
- article
- Field-Weighted Citation Impact
- 0.00