Strategy-Switch: From All-Reduce to Parameter Server for Faster Efficient Training

Deep learning plays a pivotal role in numerous big data applications by enhancing the accuracy of models. However, the abundance of available data presents a challenge when training neural networks on a single node. Consequently, various distributed training methods have emerged. Among these, two prevalent approaches are <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">All-Reduce</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Parameter Server</i> . <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">All-Reduce</i> , operating synchronously, faces synchronization-related bottlenecks, while the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Parameter Server</i> , often used asynchronously, can potentially compromise the model’s performance. To harness the strengths of both setups, we introduce <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Strategy-Switch</i> , a hybrid approach that offers the best of both worlds, combining speed with efficiency and high-quality results. This method initiates training under the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">All-Reduce</i> system and, guided by an empirical rule, transitions to asynchronous <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Parameter Server</i> training once the model stabilizes. Our experimental analysis demonstrates that we can achieve comparable accuracy to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">All-Reduce</i> training but with significantly accelerated training.