Sub-Second Model Activation for Multi-Tenant LLM Serving

Organizations that serve many fine-tuned and specialized large language models (LLMs) typically dedicate at least one GPU to each model, because moving a model onto a GPU from a cold start takes tens of seconds. Most of those GPUs sit idle most of the time. We present Nodus, a serving system whose central capability is model activation: changing which model a GPU serves in 0.161–0.766 s (p50), versus 34.5–49.9 s for a cold start (45–300× faster), with bit-exact tensors and deterministic outputs. Nodus also suspends idle tenants to a small resident footprint (~1 GB while suspended; 4,529 MiB in a combined end-to-end test) and resumes them in 0.44–2.77 s for 8B–120B models, with 25/25 outputs matching the reference. Because activation is cheap, it can be done often: on a 32-tenant replayed workload on one 8×H100 node, Nodus performed 176–203 model re-assignment events per 15 minutes at p95 0.39 s, and served all tenants at SLO on 4 GPUs instead of 32 dedicated GPUs (8× fewer), with 7.6–8.0× more SLO-met requests per GPU-second and 7.8–8.1× lower cost per SLO-met token across three traffic seeds and zero errors. At 2× traffic, a self-planned 5-GPU layout met SLO against 32 dedicated GPUs (6.4× fewer).