Hierarchical Communication Optimization for Distributed DNN Training

Motivation: Large scale deep neural network models use a lot of training data, evident from the large datasets curated for their training, such as ImageNet. Thus it is necessary to use high performance computing (HPC) systems for training of these large-scale models. Various techniques have been developed for parallelizing model training - data-parallelism, model-parallelism, and the more recent FSDP (Fully Sharded Data Parallel).