xbat: a continuous benchmarking tool for HPC software

Benchmarking the performance of one’s application in high performance computing (HPC) systems is critically important for reducing runtime and energy costs. Yet, accessing the plethora of relevant metrics that impact performance is often challenging, particularly for users without hardware experience. In this paper, we introduce the novel benchmarking tool xbat developed by MEGWARE GmbH. xbat requires no setup from the user side, and it allows the user to run, monitor and evaluate their application from the tool’s web interface, consolidating the entire benchmarking process in an approachable, intuitive workflow. We demonstrate the capabilities of the tool using benchmark applications of varying complexity and show that it can manage all aspects of the benchmarking workflow in a seamless manner. In particular, we focus on the open-source molecular dynamics research software ls1 mardyn, and the closed-source optimisation package Gurobi. Both packages present unique challenges. Mixed-integer programming solvers, such as those integrated in the Gurobi software, exhibit significant performance variability, so that seemingly innocuous parameter changes and machine characteristics can affect the runtime drastically, and ls1 mardyn comes with an auto-tuning library AutoPas, that enables the selection of various node-level algorithms to compute molecular trajectories. Focusing on these two packages, we showcase the practicality, versatility and utility of xbat, and share its current and future developments.