Predictive Reliability and Calibration of Workload Autoscaling Models for Large Language Model Inference Clusters Under Sparse Telemetry and Runtime Anomalies
Abstract Autoscalers for large language model (LLM) inference clusters that add capacity when a predicted overload probability crosses a threshold depend on the probability scale, not only on the ranking. This simulation study evaluates five such predictors under sparse telemetry and runtime anomalies on two latent-state generators and a request-level serving simulator with queueing, service-level objectives (SLOs), startup delay, and modeled power. With 3-period instead of 50-period histories, models trained on long histories became overconfident (expected calibration error up to 0.099) or conservative, whereas a Markov filter stayed calibrated and retraining on short histories removed most of the error. Temperature scaling fitted on long histories did not transfer; refitted for short histories, it cut this error by up to 68.7%, removed all 290 wrong high-confidence predictions of one model by lowering their confidence, and cut the overconfident models' idealized unmitigated overload by 59.8–82.8% at the cost of more idle capacity. With a 20-s startup delay in the request-level simulator, learned predictors used 24.0–25.3% less modeled energy than always-on scale-out, with a 1.0–1.2 percentage-point increase in the SLO-violation rate, and 95.3% of scale-out-induced violations occurred on slow new replicas. Calibration should therefore be validated per telemetry regime and checked against request-level outcomes.
Authors
- Mingxuan Tao (ORCID: https://orcid.org/0009-0007-5055-8247)
Institutions
- Emporia State University (US)
Publication Details
- Journal
- Research Square
- Published
- 2026-11-13
- DOI
- https://doi.org/10.21203/rs.3.rs-11706225/v1
- Primary Topic
- Cloud Computing and Resource Management
- Type
- preprint