Right-Sizing Multi-Tenant LLM Fleets: Self-Planned GPU Consolidation from Observed Traffic

Organizations that serve many fine-tuned or specialized large language models (LLMs) commonly give each tenant its own GPU. This is simple and isolates tenants, but most of the fleet sits idle most of the time. We study self-planned GPU consolidation: given a set of tenants and their observed traffic, the serving system itself chooses how many GPUs to use and how to lay tenants out on them, subject to a per-class latency service-level objective (SLO) gate. On a single 8× H100 SXM node serving 32 synthetic tenants in 4 classes (full fine-tunes of an 8B model, separate-architecture models, LoRA adapters on a shared 8B base, and a 32B model served in FP8), replaying the same traffic to both arms, the consolidated deployment needs 4 GPUs at 1× traffic (the measured floor; 2 GPUs fail the gate) versus 32 dedicated GPUs. Across 3 traffic seeds this is 8× fewer GPUs and 7.8×–8.1× lower cost per SLO-met token, with 0 errors and a lowest per-class SLO attainment of 98.5%. When traffic is doubled, the system self-plans a 5-GPU layout versus 32 dedicated GPUs (6.4× fewer GPUs, 6.4× lower cost per SLO-met token, 99.98% SLO attainment, 0 errors; single seed). GPU demand therefore grows sub-linearly with traffic in the measured range, at the price of higher but in-SLO time-to-first-token and time-per-output-token. At 3× traffic the system self-plans 6 GPUs versus 32 dedicated (5.333× fewer GPUs, 5.384× lower cost per SLO-met token, lowest per-class SLO attainment 0.9978, 0 errors; single seed).