Integrated Continuous Benchmarking: A pitch to extend testing to track performance

On December 12th 2024, Dirk Brömmel from Jülich Supercomputing Centre (Forschungszentrum Jülich) gave a talk in the HiRSE Seminar about Integrated Continuous Benchmarking. Abstract: When developing software, it is often relevant to track its performance. As foundation for research where available time can be a limiting factor, this should hold even more true for research software. It is even vital when targeting high-performance computing (HPC). Since changes to the software itself, the used toolchains, or the system setup should not compromise how fast users obtain their results, benchmarking should be an integral part of testing, in particular for HPC codes. Ideally, performance or scalability should only ever increase. At the same time, up-to-date benchmarks that are publicly available can advertise the code and inform users how to set-up the software in the most ideal way or whether they are achieving the expected performance. To limit the burden on developers, the aforementioned steps should be automated within continuous integration (CI) practices, introducing continuous benchmarking (CB) to it. We show our approach that works standalone without additional services and combines results over time to enable further processing or for the generation of webpages. Via a fully authenticated runner on HPC systems, it also adds the possibility of benchmarks up to system scale. Learn more about the HiRSE Seminar Series: https://www.helmholtz-hirse.de/series.html