A Fine-Grained Trace Analysis of GPU-initiated Halo Exchange in GROMACS

Strong scaling of molecular dynamics simulations in GROMACS is often limited by halo-exchange communication. Traditional CPU-centric communication approaches stall the GPU, hindering communication-computation overlap. Prior research (Doijade et al., 2025) introduced a GPU-initiated NVSHMEM redesign, fusing data packing and communication into GPU kernels. This demonstrated significant strong-scaling gains (up to 1.5× intra-node, 2× multi-node) based on end-to-end timings. This poster extends that work by providing a granular, low-level trace analysis to dissect how these optimizations contribute to performance at a microsecond level. We employed GPU-side instrumented tracing, using special registers to capture high-resolution timelines of the simulation loop. This methodology allows us to visualize and quantify the interplay between compute, data packing, and asynchronous data transfers, revealing details of concurrent kernel scheduling and overlap. Our trace analysis provides direct, visual evidence of the mechanisms, contrasting MPI's GPU idle time with NVSHMEM's successful overlap. We quantify the reduction in critical-path stalls and demonstrate the underlying mechanisms for previously observed behaviors. This low-level analysis definitively validates how the GPU-initiated NVSHMEM design achieves its gains.