Cooperative Two-Dimensional Hierarchical All-Reduce for Large Language Model Training with Heterogeneous Communication Links

Parallel and distributed learning frameworks are now widely employed for the training of large language models. However, the extremely large model size and data size have made the communication a bottleneck, severely reducing the training efficiency. In this paper, we introduce two novel hierarchical allreduce algorithms designed for 2D-torus topology. By exploiting the cooperation of the intra-cluster communication link and the inter-cluster communication link, the congestion bottleneck caused by the imbalanced link bandwidth can be alleviated. Numerical results are presented to demonstrate the performance of the proposed algorithms in comparison with existing baseline algorithms.