From In-Network Allreduce to in-Nic Allreduce in Support of Optically Switched Gpu Networks

We propose in-NIC AllReduce to lower communication times, adapting to optically switched GPU network for distributed training. The experimental results show a <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.89 \times$</tex> acceleration aligning with a theoretical analysis of <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.90 \times$</tex> acceleration on average.