Time-Varying Microservice Orchestration With Routing for Dynamic Call Graphs via Multi-Scale Deep Reinforcement Learning

Lightweight microservices as a software architecture have been widely adopted in online application development. However, in highly-concurrent microservice scenarios, frequent data communications, complex call dependencies, and dynamic delay requirements bring great challenges to efficient microservice orchestration. In this case, service deployment and request routing are interactively-coupled in multi-instance modeling, and cannot be locally optimized effectively, thereby enlarging the difficulty for collaborative orchestration. To accommodate time-varying request properties and dynamic microservice multiplexing, orchestration schemes are frequently adapted to real-time parallel request queues, further complicating the difficulty. Nevertheless, most previous work failed to propose appropriate models and methods for the above issues. Therefore, this paper investigates the online microservice orchestration with probabilistic routing for dynamic call graphs in clouds. First, we formulate the time-slot-based joint optimization problem as a Markov Decision Process. The open Jackson queuing networks are used to accurately establish multi-instance models and analyze the request queuing, processing, and communicating delays. Then, we propose an efficient curiosity-driven deep reinforcement learning algorithm, which meticulously implements instance-level orchestration through multi-dimensional collaborative decisions and multi-time-scale trigger events. Finally, through comprehensive trace-driven experiments, our proposed approach significantly outperforms other baselines in terms of orchestration cost and resource utilization.