References of "Harman, Mark"
     in
Bookmark and Share    
Full Text
Peer Reviewed
See detailAre mutants really natural? A study on how “naturalness” helps mutant selection
Jimenez, Matthieu UL; Titcheu Chekam, Thierry UL; Cordy, Maxime UL et al

in Proceedings of 12th International Symposium on 
 Empirical Software Engineering and Measurement (ESEM'18) (2018, October 11)

Background: Code is repetitive and predictable in a way that is similar to the natural language. This means that code is ``natural'' and this ``naturalness'' can be captured by natural language modelling ... [more ▼]

Background: Code is repetitive and predictable in a way that is similar to the natural language. This means that code is ``natural'' and this ``naturalness'' can be captured by natural language modelling techniques. Such models promise to capture the program semantics and identify source code parts that `smell', i.e., they are strange, badly written and are generally error-prone (likely to be defective). Aims: We investigate the use of natural language modelling techniques in mutation testing (a testing technique that uses artificial faults). We thus, seek to identify how well artificial faults simulate real ones and ultimately understand how natural the artificial faults can be. %We investigate this question in a fault revelation perspective. Our intuition is that natural mutants, i.e., mutants that are predictable (follow the implicit coding norms of developers), are semantically useful and generally valuable (to testers). We also expect that mutants located on unnatural code locations (which are generally linked with error-proneness) to be of higher value than those located on natural code locations. Method: Based on this idea, we propose mutant selection strategies that rank mutants according to a) their naturalness (naturalness of the mutated code), b) the naturalness of their locations (naturalness of the original program statements) and c) their impact on the naturalness of the code that they apply to (naturalness differences between original and mutated statements). We empirically evaluate these issues on a benchmark set of 5 open-source projects, involving more than 100k mutants and 230 real faults. Based on the fault set we estimate the utility (i.e. capability to reveal faults) of mutants selected on the basis of their naturalness, and compare it against the utility of randomly selected mutants. Results: Our analysis shows that there is no link between naturalness and the fault revelation utility of mutants. We also demonstrate that the naturalness-based mutant selection performs similar (slightly worse) to the random mutant selection. Conclusions: Our findings are negative but we consider them interesting as they confute a strong intuition, i.e., fault revelation is independent of the mutants' naturalness. [less ▲]

Detailed reference viewed: 77 (12 UL)
Full Text
Peer Reviewed
See detailAn Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation that Avoids the Unreliable Clean Program Assumption
Titcheu Chekam, Thierry UL; Papadakis, Mike UL; Le Traon, Yves UL et al

in International Conference on Software Engineering (ICSE 2017) (2017, May 28)

Many studies suggest using coverage concepts, such as branch coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between coverage ... [more ▼]

Many studies suggest using coverage concepts, such as branch coverage, as the starting point of testing, while others as the most prominent test quality indicator. Yet the relationship between coverage and fault-revelation remains unknown, yielding uncertainty and controversy. Most previous studies rely on the Clean Program Assumption, that a test suite will obtain similar coverage for both faulty and fixed (‘clean’) program versions. This assumption may appear intuitive, especially for bugs that denote small semantic deviations. However, we present evidence that the Clean Program Assumption does not always hold, thereby raising a critical threat to the validity of previous results. We then conducted a study using a robust experimental methodology that avoids this threat to validity, from which our primary finding is that strong mutation testing has the highest fault revelation of four widely-used criteria. Our findings also revealed that fault revelation starts to increase significantly only once relatively high levels of coverage are attained. [less ▲]

Detailed reference viewed: 303 (34 UL)
Full Text
Peer Reviewed
See detailDetecting Trivial Mutant Equivalences via Compiler Optimisations
Kintis, Marinos UL; Papadakis, Mike UL; Jia, Yue et al

in IEEE Transactions on Software Engineering (2017)

Detailed reference viewed: 98 (4 UL)
Full Text
Peer Reviewed
See detailMutation Testing Advances: An Analysis and Survey
Papadakis, Mike UL; Kintis, Marinos UL; Zhang, Jie et al

in Advances in Computers (2017)

Detailed reference viewed: 791 (44 UL)
Full Text
Peer Reviewed
See detailThreats to the validity of mutation-based test assessment
Papadakis, Mike UL; Henard, Christopher; Harman, Mark et al

in International Symposium on Software Testing and Analysis, ISSTA 2016 (2016)

Detailed reference viewed: 78 (8 UL)
Full Text
Peer Reviewed
See detailComparing White-box and Black-box Test Prioritization
Henard, Christopher UL; Papadakis, Mike UL; Harman, Mark et al

in 38th International Conference on Software Engineering (ICSE'16) (2016)

Although white-box regression test prioritization has been well-studied, the more recently introduced black-box prioritization approaches have neither been compared against each other nor against more ... [more ▼]

Although white-box regression test prioritization has been well-studied, the more recently introduced black-box prioritization approaches have neither been compared against each other nor against more well-established white-box techniques. We present a comprehensive experimental comparison of several test prioritization techniques, including well-established white-box strategies and more recently introduced black-box approaches. We found that Combinatorial Interaction Testing and diversity-based techniques (Input Model Diversity and Input Test Set Diameter) perform best among the black-box approaches. Perhaps surprisingly, we found little difference between black-box and white-box performance (at most 4% fault detection rate difference). We also found the overlap between black- and white-box faults to be high: the first 10% of the prioritized test suites already agree on at least 60% of the faults found. These are positive findings for practicing regression testers who may not have source code available, thereby making white-box techniques inapplicable. We also found evidence that both black-box and white-box prioritization remain robust over multiple system releases. [less ▲]

Detailed reference viewed: 180 (8 UL)
Full Text
Peer Reviewed
See detailTrivial Compiler Equivalence: A Large Scale Empirical Study of a Simple, Fast and Effective Equivalent Mutant Detection Technique
Papadakis, Mike UL; Yue, Jia; Harman, Mark et al

in 37th International Conference on Software Engineering (ICSE 2015) (2015)

Detailed reference viewed: 86 (3 UL)
Full Text
Peer Reviewed
See detailCombining Multi-Objective Search and Constraint Solving for Configuring Large Software Product Lines
Henard, Christopher UL; Papadakis, Mike UL; Harman, Mark et al

in 37th International Conference on Software Engineering (ICSE 2015) (2015)

Detailed reference viewed: 161 (9 UL)
Full Text
Peer Reviewed
See detailAutomated Generation of State Abstraction Functions using Data Invariant Inference
Nguyen, Duy Cu UL; Tonella, Paolo; Marchetto, Alessandro et al

Scientific Conference (2013, May 18)

Detailed reference viewed: 66 (2 UL)