Reproducible and hackable software benchmarking with(out) compute clusters

Abstract We present Simsala, an easy-to-install and easy-to-use collection of scripts supporting benchmarking on clusters of compute nodes. While designed with applications for benchmarking solving technologies like SAT and extensions, our solution can be easily transferred to many other application scenarios as well. In this work, we discuss the design objectives behind Simsala, provide some implementation details, and illustrate with case studies how Simsala has been applied in the past for extensive evaluations.