Artifact Description: An Evaluation of the Effect of Network Cost Optimization for Leadership Class Supercomputers

An Evaluation of the Effect of Network Cost Optimization for Leadership Class Supercomputers Overview This repository contains the experimental artifacts, scripts, and benchmarks used in our SC paper titled "An Evaluation of the Effect of Network Cost Optimization for Leadership Class Supercomputers." The paper presents a detailed analysis and comparison of network topology costs and performance between the Frontier and Summit supercomputers at Oak Ridge National Laboratory (ORNL). This repository specifically focuses on the artifacts related to C3, including the configurations, scripts, and results of various benchmarks and applications. Benchmarks and Applications The following benchmarks and applications were used to evaluate the network topologies of Frontier and Summit: All-to-All Microbenchmark: A custom micro-benchmark that generates an all-to-all communication pattern across the full system. MPIGraph: A pair-wise point-to-point communication benchmark that measures the available bandwidth between every node pair on the system. HALO3D: A MOTIF-based benchmark that generates a nearest-neighbor communication pattern. M-PSDNS: Part of the OLCF-6 benchmark suite, focusing on 3D FFTs and fluid turbulence simulation. LAMMPS: A molecular dynamics application benchmark, also part of the OLCF-6 benchmark suite. AthenaPK: An astrophysical magnetohydrodynamics code that includes Kokkos and adaptive mesh refinement (AMR) support. Flash-X Maclaurin Spheroid: A multi-physics code used for simulating astrophysical scenarios, focusing on all-reduce communication patterns. Node Allocation Policies/Layouts To ensure a comprehensive evaluation, we executed each application using three node allocation policies/layouts: Default: The default layout generated by the job scheduler during normal production operations. Block: A layout that maximizes locality according to the system-specific network topology, executed on an idle system. Random: A layout that disrupts node locality by randomly shuffling nodes within neighboring MPI ranks across dragonfly groups. Scripts for generating these layouts on both Frontier and Summit are included in this repository. Scripts and Configurations Node Allocation Scripts: Scripts for block and random node allocation on both Frontier and Summit. Application-Specific Configurations: Job scripts, input files, and environment setup commands for each application used in the experiments. Benchmark Execution: Commands and scripts for running each benchmark, with configurations tailored to stress the network topologies on Frontier and Summit. Methodology and Experimentation The results presented in the paper are based on experiments conducted on the Frontier and Summit systems. These systems feature distinct network topologies: a dragonfly topology for Frontier and a fat-tree topology for Summit. Experiment Setup Frontier Architecture: HPE Cray EX 235a “Bard Peak” node architecture with AMD EPYC 7A53 “Trento” CPUs, AMD MI250X GPUs, and Slingshot interconnect. Summit Architecture: IBM AC922 node architecture with IBM POWER9 processors, NVIDIA V100 GPUs, and two ports of EDR InfiniBand. Data Collection Figures of Merit (FOM): For each application, we collected reports of the FOM to determine the scaling efficiency. Per Rank and Per Node Bandwidth: The experimental results include per rank bandwidth, per node bandwidth, and speedup, measured across different workloads and communication patterns. Acknowledgments This work was supported by the Oak Ridge Leadership Computing Facility at ORNL, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Contact For any questions or issues, please contact Awais Khan or Jack Lange ([email protected], [email protected])