Impact of JVM Configurations on Test Runtime

JVM provides dozens of configuration flags, with many flags intended for tuning application performance. We empirically study the impact of JVM configuration flags on software testing runtime. We focus on an extensive study that shows not only the great impact of JVM configurations on test runtime (up to 43.89% reduction in runtime when using certain configurations) but also shows that those configurations that reduce runtime are rare and thus hard to find. Modern techniques based on machine learning or combinatorial testing that search through combinations of configuration flags are still not as effective at finding the best configurations for test runtime. Finally, we show that JVM configurations that provide good speedup retain this power over a number of commits. We believe that this paper provides strong motivation for further work on finding the best JVM configurations to optimize test runtime.