Research on Optimization of Communication Technology Based on Deep Learning

Distributed deep learning systems commonly use synchronous data parallelism to train models. However, communication overhead can be costly in distributed environments with limited communication bandwidth. To reduce communication overhead, an intuitive approach is to increase the communication interval. However, increasing the communication interval often affects the convergence speed of the model, requiring more epochs to train the model to the target accuracy, thereby reducing the statistical efficiency of the training algorithm. To address these issues in distributed synchronous data parallel training algorithms and communication interval selection methods, this paper proposes a training algorithm that combines skip communication strategies and correction techniques to ensure low communication overhead and high statistical efficiency. The training algorithm maintains a small batch size through local updates in each training process and employs correction techniques to reduce the divergence among individual local models, thereby ensuring high statistical efficiency. At the same time, the training algorithm utilizes a skip communication strategy to update the global model, meaning that the global model is not updated with every iteration, but rather updated once every few iterations. This reduces the frequency of communication, ensuring low communication overhead. Experimental results show that this training algorithm reduces total training time by 88.9%, 74.9%, 34.1%, and 82.4% compared to SSGD, SkipSSGD, LocalSGD, and SMA, respectively.