SyncNet: Harmonizing Nodes for Efficient Learning

This research investigates a novel communication-efficient algorithm, SyncNet, for training large neural networks on massive datasets. The ever-growing size of datasets necessitates distributing the training workload across vast machine networks, often containing many machines. While economically feasible, such large-scale training suffers from limitations like communication overhead associated with gradient transmission and increased vulnerability to network failures. SyncNet addresses these limitations by employing a remarkably simple approach. Nodes in the network transmit only the sign (positive or negative) of their gradient vector to a central server. This server then aggregates these signs using a majority vote to determine the overall update direction for the model parameters. Compared to traditional full-precision distributed SGD, SyncNet demonstrates approximately a 25 percent reduction in per iteration time. The research establishes theoretical convergence, which guarantees for SyncNet in both large-scale and mini-batch settings under realistic conditions verified through experimentation.