Assessing model-driven mutation testing of Java bytecode

Abstract Mutation testing is an approach to checking the robustness of test suites. The program code is slightly modified by mutations to inject bugs, and a test suite is robust enough if it finds them. Mutation testing tools provide sets of mutation operators, such as swapping arithmetic operators, to make small modifications to the program. The results of mutation tests depend directly on the possible mutations. These mutations should cause actual changes in the program behavior, but also should not prevent the program from being loaded and executed. The more advanced mutations are, the more they challenge the test suite. Existing non-model-based mutation testing tools do not support the definition of advanced mutation operators that go beyond manipulating a small number of adjacent instructions within a single method. Thus, we present a model-driven approach where mutations of Java bytecode can be flexibly defined as model transformations. Our tool, Model-based Mutation Testing (MMT), implements this approach and includes model transformations for conventional and advanced mutation operators, such as deleting overridden methods or changing type casts. To evaluate the effectiveness and efficiency of model-driven mutation testing, we have applied MMT to all projects and versions in Defects4J, a well-established collection of real-world Java projects with reproducible bugs. We check for MMT’s ability to generate mutants close to real bugs and compare it with the non-model-based mutation testing tools Jumble, PIT, and $$\mu $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>μ</mml:mi> </mml:math> BERT. Our evaluation shows that MMT and PIT are significantly more effective and efficient than $$\mu $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>μ</mml:mi> </mml:math> BERT and Jumble. MMT even outperforms PIT in its ability to generate such realistic bugs, with similar efficiency per generated mutant. Jumble and $$\mu $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mi>μ</mml:mi> </mml:math> BERT are one or two orders of magnitude slower than MMT and PIT. There are some bugs reconstructed by only one of the tools, including some that only the advanced operators of MMT could replicate. Fifteen percent of the Defects4J project versions had bugs that could not be reconstructed by the mutation operators of any of the investigated tools. This shows that further research in mutation operators, as enabled by MMT, has high potential. Our mutation testing tool MMT is available online https://gitlab.uni-marburg.de/fb12/plt/modbeam-mt/mmt, as well as all the evaluation data (Ancona et al., Evaluation data of comparison of mutation testing tools. https://doi.org/10.5281/zenodo.20054492).