LOSM: Leveraging OpenMP and Shared Memory for Accelerating Blocking MPI Allreduce

This paper introduces a novel method for optimizing the performance of MPI Allreduce calls in distributed deep learning applications. Our approach maximizes efficiency by utilizing all available cores within an MPI process, both for single and multi-node blocking MPI Allreduce calls. Unlike previous methods that involve separate and exclusive core allocation for compute and communication, our approach leverages all available cores to drive both memory and network. For single-node scenarios, we employ OMP pragmas to efficiently utilize multiple cores for memory copies and reductions within shared memory. In multi-node, we leverage multiple cores to concurrently copy data into shared memory and drive the network, via multiple MPI endpoints. Our technique requires no modification to existing application binaries and demonstrates significant speedup (≥ 4X) on allreduce microbenchmark and up to 16% application performance improvement, on BERT [1] for MLPERF configurations, as compared to the baseline MPI.