Mutation Testing of Java Bytecode: A Model-Driven Approach

Mutation testing is an approach to checking the robustness of test suites. The program code is slightly changed by mutations to inject bugs. A test suite is robust enough if it finds such bugs. Mutation testing tools typically integrate sets of mutation operators such as, for example, swapping arithmetic operators; modern tools typically work with compiled code such as Java bytecode. The mutations must be defined in such a way that the mutated program can still be loaded and executed. The results of mutation tests depend directly on the possible mutations. More advanced mutations and even domain-specific mutations can pose another challenge to the test suite. Since the classical, non-model-based mutation testing tools do not support the specification of advanced mutation operators well, we propose a model-driven approach where mutations of Java bytecode can be flexibly defined by model transformation. Our approach also provides advanced mutation operators for modifying object-oriented structures, Java-specific properties and API method calls, making it the only mutation testing tool for Java bytecode that supports such mutations. To further improve the effectiveness of mutation testing, mutants are generated only for bytecode that is covered by tests. Our approach is implemented in the MMT tool. It has been evaluated against non-model-based mutation testing tools for its ability to generate mutants close to real bugs. The experiments make use of Defects4J, a well-established collection of real-world Java projects with reproducible bugs.