Pushing the Performance Boundary of In-Network AllReduce With Joint Topology and Routing Optimization

Parameter server (PS) based AllReduce is widely employed by distributed machine learning (DML) applications for model synchronization. As recent studies show, the execution of synchronization can be greatly accelerated by letting powerful switches (a.k.a., aggregators) inside the network pre-aggregate the associated traffic during the journey, yielding the solution of In-Network (accelerated) AllReduce. We i) find that the key to pushing the performance boundary of in-network AllReduce relies on controlling the system’s topology and routing jointly since associated flows must go through the same aggregator(s) for in-network acceleration, thus $i i)$ propose the solution of ATRO, i.e., Aggregator-aware Topology and Routing Optimization, to achieve this optimality. At the core, ATRO i) envisions a realistic scenario where commercially available optical switching devices are employed to build a topology-reconfigurable cluster and $i i)$ jointly manages its topology and routing with a novel theory-empowered algorithm to maximize the throughput of in-network accelerated AllReduce. As confirmed by evaluation studies, ATRO achieves excellent performances: compared to the state-of-the-art scheme that i) only optimizes the routing, $i i)$ jointly optimizes the routing and task placements, and $i i i$) jointly optimizes the routing and aggregator placements, it could accelerate the AllReduce’s execution (in terms of the achieved throughput) by up to $10.4 \times$, $22.1 \times$, and $14.0 \times$, respectively.