Sign-SGD with WFBP for Communication-Efficient Distributed Deep Learning

Distributed deep neural network (DNN) training is crucial for scaling large models but is often hindered by communication overheads during gradient aggregation. Gradient compression methods like Sign-SGD and system optimizations such as Wait-Free Backpropagation (WFBP) have been proposed to address this. This paper evaluates their combined impact on real-world DNN training. Extensive experiments on diverse models and datasets reveal that combining Sign-SGD with WFBP significantly reduces communication time without compromising model accuracy, outperforming Synchronous Stochastic Gradient Descent (S-SGD) and other baselines. The study analyzes trade-offs between compression ratios, communication overheads, and computational costs, offering practical insights into gradient compression and system optimization integration. These findings highlight the potential of Sign-SGD and WFBP to enhance distributed training, particularly in bandwidth-constrained environments, accelerating development and enabling efficient large-scale learning while maintaining performance.