Experiment runner: A tool for the automatic orchestration of experiments targeting software systems

Experiment Runner (ER) is a Python framework for the (semi-) automatic orchestration of measurement-based experiments targeting software systems. ER allows users to (i) programmatically define an experiment in terms of its factors, treatments, subjects, etc., (ii) start, stop, pause, and resume an experiment with no data loss, (iii) integrate third-party software- or hardware-profilers, (iv) execute an experiment either in automatic mode or semi-automatic mode, and (v) define custom callbacks that are automatically called by ER when needed. ER has been used in tens of empirical studies in the Software Engineering area. In this paper, we describe the main features of ER, its software architecture, a simple experiment executed by ER, and three scientific studies where it has been used successfully.