Maestro: Learning-Based Reuse of Activated Experts to Mitigate Long-Tail Latency under Heavy MoE Workloads in Cloud

Cloud LLM services increasingly adopt Mixture-of-Experts (MoE) models for cost-effective scaling, yet under heavy multi-task workloads their serving performance is often bottlenecked by severe tail latency. The execution semantics of MoE inference are poorly aligned with cloud-native serving abstractions, causing spatial and temporal mismatch in expert placement and demand, and consequently frequent expert loading. Expert demand, however, exhibits structured locality where correlated requests and decoding steps often induce overlapping expert working sets, creating opportunities for activated-expert reuse. Exploiting such reuse is non-trivial because request routing, expert reuse, and resource allocation are tightly coupled over time, forming a sequential decision problem whose coupled dimensions existing systems address only in isolation. This paper presents Maestro, a learning-based framework for activated-expert reuse in cloud-native MoE inference. Maestro models tasks, experts, instances, and execution stages as a heterogeneous graph with relational message passing to capture structured system state, and combines a lightweight Neural Optimistic Value Policy (NOVP) with Pareto-guided action ranking (AdaFilter) for efficient online decision making. It further introduces a prefill-driven expert prefetch mechanism that overlaps loading with computation, where the per-window transfer budget is jointly determined by the RL controller to balance reuse coverage against memory pressure. Evaluated on Mixtral-8 × 7B, Phi-3.5-MoE, and Qwen1.5-MoE, Maestro reduces P50 and P99 latency by up to 34.02% and 28.94%, respectively, and achieves up to 1.51 × the throughput of vLLM. Against the SOTA learning-based methods, it reduces P99 and TTFT by 8.58%–17.47% and 6.22%–17.43%, respectively, while improving throughput by 12.09%–22.70%.

Authors

Institutions

Publication Details

Journal
ACM Transactions on Architecture and Code Optimization
Published
2026-11-13
DOI
https://doi.org/10.1145/3862593
Primary Topic
Cloud Computing and Resource Management
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Maestro: Learning-Based Reuse of Activated Experts to Mitigate Long-Tail Latency under Heavy MoE Workloads in Cloud

Danghui Wang, Jihe Wang, Jianfeng An, Linying Wu et al.
ACM Transactions on Architecture and Code Optimization
Cloud Computing and Resource Management
article

Maestro: Learning-Based Reuse of Activated Experts to Mitigate Long-Tail Latency under Heavy MoE Workloads in Cloud

Danghui Wang, Jihe Wang, Jianfeng An, Linying Wu, Yuanhua Xiao, Zhiyu Sun, Yulu Liu
article en

Abstract

Cloud LLM services increasingly adopt Mixture-of-Experts (MoE) models for cost-effective scaling, yet under heavy multi-task workloads their serving performance is often bottlenecked by severe tail latency. The execution semantics of MoE inference are poorly aligned with cloud-native serving abstractions, causing spatial and temporal mismatch in expert placement and demand, and consequently frequent expert loading. Expert demand, however, exhibits structured locality where correlated requests and decoding steps often induce overlapping expert working sets, creating opportunities for activated-expert reuse. Exploiting such reuse is non-trivial because request routing, expert reuse, and resource allocation are tightly coupled over time, forming a sequential decision problem whose coupled dimensions existing systems address only in isolation. This paper presents Maestro, a learning-based framework for activated-expert reuse in cloud-native MoE inference. Maestro models tasks, experts, instances, and execution stages as a heterogeneous graph with relational message passing to capture structured system state, and combines a lightweight Neural Optimistic Value Policy (NOVP) with Pareto-guided action ranking (AdaFilter) for efficient online decision making. It further introduces a prefill-driven expert prefetch mechanism that overlaps loading with computation, where the per-window transfer budget is jointly determined by the RL controller to balance reuse coverage against memory pressure. Evaluated on Mixtral-8 × 7B, Phi-3.5-MoE, and Qwen1.5-MoE, Maestro reduces P50 and P99 latency by up to 34.02% and 28.94%, respectively, and achieves up to 1.51 × the throughput of vLLM. Against the SOTA learning-based methods, it reduces P99 and TTFT by 8.58%–17.47% and 6.22%–17.43%, respectively, while improving throughput by 12.09%–22.70%.

ACM Transactions on Architecture and Code Optimization
Northwestern Polytechnical University (CN)
Cloud Computing and Resource Management
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon