Predictive Reliability and Calibration of Workload Autoscaling Models for Large Language Model Inference Clusters Under Sparse Telemetry and Runtime Anomalies

Abstract Autoscalers for large language model (LLM) inference clusters that add capacity when a predicted overload probability crosses a threshold depend on the probability scale, not only on the ranking. This simulation study evaluates five such predictors under sparse telemetry and runtime anomalies on two latent-state generators and a request-level serving simulator with queueing, service-level objectives (SLOs), startup delay, and modeled power. With 3-period instead of 50-period histories, models trained on long histories became overconfident (expected calibration error up to 0.099) or conservative, whereas a Markov filter stayed calibrated and retraining on short histories removed most of the error. Temperature scaling fitted on long histories did not transfer; refitted for short histories, it cut this error by up to 68.7%, removed all 290 wrong high-confidence predictions of one model by lowering their confidence, and cut the overconfident models' idealized unmitigated overload by 59.8–82.8% at the cost of more idle capacity. With a 20-s startup delay in the request-level simulator, learned predictors used 24.0–25.3% less modeled energy than always-on scale-out, with a 1.0–1.2 percentage-point increase in the SLO-violation rate, and 95.3% of scale-out-induced violations occurred on slow new replicas. Calibration should therefore be validated per telemetry regime and checked against request-level outcomes.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11706225/v1
Primary Topic
Cloud Computing and Resource Management
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Predictive Reliability and Calibration of Workload Autoscaling Models for Large Language Model Inference Clusters Under Sparse Telemetry and Runtime Anomalies

Mingxuan Tao
1 links
Research Square
Cloud Computing and Resource Management
preprint

Predictive Reliability and Calibration of Workload Autoscaling Models for Large Language Model Inference Clusters Under Sparse Telemetry and Runtime Anomalies

Mingxuan Tao
preprint en

Abstract

Abstract Autoscalers for large language model (LLM) inference clusters that add capacity when a predicted overload probability crosses a threshold depend on the probability scale, not only on the ranking. This simulation study evaluates five such predictors under sparse telemetry and runtime anomalies on two latent-state generators and a request-level serving simulator with queueing, service-level objectives (SLOs), startup delay, and modeled power. With 3-period instead of 50-period histories, models trained on long histories became overconfident (expected calibration error up to 0.099) or conservative, whereas a Markov filter stayed calibrated and retraining on short histories removed most of the error. Temperature scaling fitted on long histories did not transfer; refitted for short histories, it cut this error by up to 68.7%, removed all 290 wrong high-confidence predictions of one model by lowering their confidence, and cut the overconfident models' idealized unmitigated overload by 59.8–82.8% at the cost of more idle capacity. With a 20-s startup delay in the request-level simulator, learned predictors used 24.0–25.3% less modeled energy than always-on scale-out, with a 1.0–1.2 percentage-point increase in the SLO-violation rate, and 95.3% of scale-out-induced violations occurred on slow new replicas. Calibration should therefore be validated per telemetry regime and checked against request-level outcomes.

Research Square
Emporia State University (US)
Cloud Computing and Resource Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon