How effective are mutation testing tools? An empirical analysis of Java mutation testing tools with manual analysis and real faults

Mutation analysis is a well-studied, fault-based testing technique. It requires
testers to design tests based on a set of artificial defects. The defects help
in performing testing activities by measuring the ratio that is revealed by the
candidate tests. Unfortunately, applying mutation to real-world programs
requires automated tools due to the vast number of defects involved. In such a
case, the effectiveness of the method strongly depends on the peculiarities of
the employed tools. Thus, when using automated tools, their implementation
inadequacies can lead to inaccurate results. To deal with this issue, we
cross-evaluate four mutation testing tools for Java, namely PIT, muJava, Major
and the research version of PIT, PITRV, with respect to their fault-detection
capabilities. We investigate the strengths of the tools based on: a) a set of
real faults and b) manual analysis of the mutants they introduce. We find that
there are large differences between the tools’ effectiveness and demonstrate
that no tool is able to subsume the others. We also provide results indicating
the application cost of the method. Overall, we find that PITRV achieves the
best results. In particular, PITRV outperforms the other tools by finding 6%
more faults than the other tools combined.