Is Error-Free Communication Essential in Data Centers for AI Training?

Data centers (DCs) are now heavily involved in large-scale artificial intelligence (AI) learning processes. Fiber optic systems are the primary communication medium for enabling DC networks due to their high speed and low energy consumption per bit. Traditionally, fiber optic systems are designed to ensure nearly error-free communication, which increases their power consumption and latency. In this work, we revisit the communication requirements in DCs for AI training by analytically examining how communication channel noise affects the learning process. Our analysis shows that channel errors introduce bounded noise in the model weights, which can be maintained within an acceptable range. Therefore, relaxing the error-free requirement has a negligible impact on AI learning performance while significantly improving power efficiency and latency. Additionally, we propose an optimized stack layer for the optical communication link with minimal modifications to the IEEE 802.3bs standard, tailored for AI training. To validate our approach, we simulate the training of the ChatGPT-2 model on a DC with 32 workers and noisy communication links, showing that the training process tolerate a bit error rate of up to <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1 \mathrm{e}-4$</tex>.