Ltfc: Loss-Tolerant Flow Control with RDMA Network for Machine Learning Clusters

Current AI training clusters widely use RoCEv2 to improve the communication efficiency of the interconnect networks across machines. RoCEv2 relies on Priority Flow Control (PFC) to ensure a lossless network. However, PFC brings certain side effects, such as head-of-line blocking, congestion spreading, and deadlock. Numerous studies have been proposed to eliminate the side effects. However, unlike traditional high-performance computing applications, distributed machine learning (DML) is not 100 % loss-intolerant. In light of this observation, this paper proposes a loss-tolerant flow control (LTFC). LTFC does not rely on PFC to ensure a lossless environment, but to control packet loss ratio within the tolerance threshold. Compared to the traditional trigger condition, LTFC reduces the likelihood that the PFC will be triggered. Additionally, we replace RoCEv2's default Go-back-N mechanism with a non-retransmission mechanism to eliminate retransmission latency. We demonstrate the bounded-loss tolerance feature of DML on our testbed and evaluate the performance of LTFC in large-scale simulations. Simulation results show that LTFC reduces the average flow completion time (FCT) by up to 26.9 % and tail FCT by up to 16.9 % compared to existing solutions.