Optimizing Training Performance via Omni-Block Sparsity Adaptation

In recent years, neural networks have become more complicated, there has been an increased requirement for efficient communication in distributed deep learning. Optimizing both accuracy and model training speed in traditional approaches often be hard. In this paper, we propose a novel approach called Omni-Block Sparsity Adaptation (OBSA), which partitions the model’s gradients into multiple blocks and dynamically adjusts the sparsity of each block based on its properties. Experiments demonstrate that our approach significantly improves Top-1 accuracy across multiple architectures, achieving impressive speedups of up to $11.29 x$ compared to existing AllReduce methods.