Predicting LLM Inference Server Request Capacity
Due to the growing number of applications utilizing large language models, understanding hardware requirements becomes imperative to ensure efficient operation of costly computational accelerators. This research investigates the queuing behavior of an LLM inference server as it operates near a steady-state regime. It is shown that the request capacity can be modeled as a saturation function for a constant arrival rate. Furthermore, a modified exponential function is demonstrated to predict the saturation value for the number of requests for a given arrival rate.
Authors
- Albert Reuther (ORCID: https://orcid.org/0000-0002-3168-3663)
- Michael E. Houle (ORCID: https://orcid.org/0000-0001-8486-8015)
- William Arcand
- Antonio De Rosa (ORCID: https://orcid.org/0000-0002-1392-4881)
- Julie Mullen
- William Bergeron
- David Bestor (ORCID: https://orcid.org/0009-0002-7684-1191)
- Piotr Łuszczek (ORCID: https://orcid.org/0000-0002-0089-6965)
- Daniel J. Burrill
- Matthew Hubbell
- Chansup Byun (ORCID: https://orcid.org/0009-0003-0183-914X)
- Guillermo Morales
- Michael G. Jones (ORCID: https://orcid.org/0000-0002-5434-4904)
- Vijay Gadepally (ORCID: https://orcid.org/0009-0004-5782-7137)
- Andrew Prout (ORCID: https://orcid.org/0000-0002-4408-0247)
- Larry W. Anderson
- Charles Yee
- Peter Michaleas
- Alex Bonn
Institutions
- MIT Lincoln Laboratory (US)
Publication Details
- Published
- 2025-09-15
- DOI
- https://doi.org/10.1109/hpec67600.2025.11196629
- Primary Topic
- Software System Performance and Reliability
- Type
- article
- Field-Weighted Citation Impact
- 0.00