Quality-aware cost minimization for multi-tier LLM routing in heterogeneous data centers

Abstract The rapid expansion of Large Language Model (LLM) inference workloads poses a significant challenge for cloud service providers by increasing operational costs through intensive computational demand. Traditional resource provisioning strategies typically assume homogeneous hardware and static service levels, failing to exploit the economic efficiencies available in heterogeneous data center environments. In this paper, we propose HARP (Heterogeneity-Aware Routing and Provisioning), a cost optimization framework that minimizes operational expenditure for LLM inference services. We introduce a multi-tier quality adaptation strategy that dynamically routes requests across heterogeneous GPU instance types, from legacy to state-of-the-art, under varying Quality of responses (QoR) constraints. We formalize the problem as a Mixed-Integer Linear Program (MILP) that balances the trade-offs between hardware lease costs and throughput characteristics across GPU tiers. By decoupling strict hardware dependencies from service requirements, HARP allows for intelligent offloading of lower-tier requests to more cost-effective hardware without violating Service Level Objectives (SLOs). Under a throughput estimation model combining public benchmarks with theoretical memory bandwidth scaling estimates, simulations using workload traces and on-demand pricing profiles show up to $$\sim $$ 23% lower modeled operational cost than a QoR-cost greedy heuristic (QCGH) routing baseline, with double-digit average savings across the evaluated demand compositions. Our analysis further reveals that these gains are largest in the intermediate QoR regime and that the optimizer systematically matches request complexity to the most cost-effective GPU generation, retaining legacy hardware for lower-quality tiers while reserving premium instances for the most demanding requests, underscoring that economic efficiency stems from heterogeneous instance mixing rather than from uniformly provisioning the highest-capacity hardware.