ECRMG: An Effecient and Collaborative Routing Mechanism for Geo-Distributed Machine Learning

Geo-Distributed Machine Learning (GDML) is an emerging scenario that executes distributed training (DT) across geographically dispersed datacenters. However, GDML systems face a significant routing bottleneck problem during model synchronization over the Wide Area Network (WAN). Existing solutions typically address this issue by periodically distributing DT jobs across datacenters, but they often introduce additional signaling communication latency. Therefore, this paper proposes the ECRMG, an efficient and collaborative routing mechanism. Firstly, we formulate the complicated GDML routing problem. Then, a model set mechanism is proposed to make optimal WAN routing decisions using deep reinforcement learning (DRL), aiming to minimize the time cost of gradient aggregation across datacenters. Additionally, in the control plane, we develop an application based on the Open Network Operating System (ONOS) controller to implement ECRMG. Finally, a prototype system is built to validate ECRMG, and the experimental results show that ECRMG significantly reduces the GDML system latency.