Progress and Bandwidth Aware Partial Model Synchronization With Selective Reduce

By allowing partial instead of all workers to participate in a round of model synchronization, the recent proposal of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">partial reduce</i> provides a promising way to prevent the entire training from being blocked by straggler workers in heterogeneous data-parallel distributed training. However, its current designs are far from optimal, as it selects workers for partial model synchronization agnostic to both their training progress and available bandwidths. To address these issues, we analyze the design space and propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">selective reduce</i>. By exploring the idea of <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">waiting</i> for more workers to be ready and splitting them into partial synchronization groups, based on the state of their training progress and available bandwidths, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">selective reduce</i> could not only enlarge the scale of synchronization (i.e., the number of involved workers) thus accelerating the convergence of the training but also reduce the time cost of synchronization thus making the training iterate faster. Extensive evaluations confirm that <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">selective reduce</i> outperforms <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">partial reduce</i> and is robust to both inaccurate bandwidth estimations and unknown-in-advance runtime distributions of training computation.