Load Imbalance in HPC Applications: Improved Profiling and New Ways to Use Wasted Cycles

As the scale of High Performance Computing (HPC) applications increases, load imbalance and other inefficiencies become correspondingly more significant. In this work we extend a previously described system that reclaims idle computation time during MPI synchronization. We begin by applying a new profiler to a new target system (Stampede III) and observing application characteristics. We find that for miniFE there is immense potential for capturing wasted cycles (roughly 30%). Moreover these cycles do not appear to be caused systematically by particular ranks, but rather by a number of ranks that vary without obvious pattern. We then propose a new runtime algorithm that improves CPU cycle utilization by an additional 10–15% over the previous best result.