Delay-Constrained Dynamic Network Control with Multi-Agent Deep Reinforcement Learning

Timely delivery of delay-sensitive information over dynamic, heterogeneous networks is increasingly essential for a range of interactive applications, such as industrial automation, self-driving vehicles, and augmented reality. However, most existing network control solutions target only average delay performance, falling short of providing strict End-to-End (E2E) peak latency guarantees. This paper addresses the challenge of reliably delivering packets within application-imposed strict deadlines by leveraging recent advancements in Multi-Agent Deep Reinforcement Learning (MA-DRL). After introducing the Delay-Constrained Maximum-Throughput (DCMT) dynamic network control problem, and highlighting the limitations of current solutions, we present a novel MA-DRL network control framework that leverages a centralized routing and distributed scheduling architecture. Within the proposed Multi-Agent Deep Reinforcement Learning (MA-DRL) framework, we present a novel design approach that progressively incorporates critical networking domain insights to refine and optimize the action and state spaces of agents.Building on this approach and leveraging the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) technique, we design novel and highly efficient strategies in which centralized routing agents and distributed scheduling agents work in tandem to dynamically assign paths and manage packet transmissions based on their lifetimes, thereby optimizing timely delivery. Numerical results show that this framework significantly outperforms stochastic optimization-based approaches under high-congestion scenarios, showcasing the potential of MADDPG techniques for advanced network control.