Paper published in a journal (Scientific congresses, symposiums and conference proceedings)
Are mutants really natural? A study on how “naturalness” helps mutant selection
Jimenez, Matthieu; Titcheu Chekam, Thierry; Cordy, Maxime et al.
2018In Proceedings of 12th International Symposium on 
 Empirical Software Engineering and Measurement (ESEM'18)
Peer reviewed
 

Files


Full Text
esem18.pdf
Author preprint (6.29 MB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
Mutation testing; Fault Revelation; Language Models
Abstract :
[en] Background: Code is repetitive and predictable in a way that is similar to the natural language. This means that code is ``natural'' and this ``naturalness'' can be captured by natural language modelling techniques. Such models promise to capture the program semantics and identify source code parts that `smell', i.e., they are strange, badly written and are generally error-prone (likely to be defective). Aims: We investigate the use of natural language modelling techniques in mutation testing (a testing technique that uses artificial faults). We thus, seek to identify how well artificial faults simulate real ones and ultimately understand how natural the artificial faults can be. %We investigate this question in a fault revelation perspective. Our intuition is that natural mutants, i.e., mutants that are predictable (follow the implicit coding norms of developers), are semantically useful and generally valuable (to testers). We also expect that mutants located on unnatural code locations (which are generally linked with error-proneness) to be of higher value than those located on natural code locations. Method: Based on this idea, we propose mutant selection strategies that rank mutants according to a) their naturalness (naturalness of the mutated code), b) the naturalness of their locations (naturalness of the original program statements) and c) their impact on the naturalness of the code that they apply to (naturalness differences between original and mutated statements). We empirically evaluate these issues on a benchmark set of 5 open-source projects, involving more than 100k mutants and 230 real faults. Based on the fault set we estimate the utility (i.e. capability to reveal faults) of mutants selected on the basis of their naturalness, and compare it against the utility of randomly selected mutants. Results: Our analysis shows that there is no link between naturalness and the fault revelation utility of mutants. We also demonstrate that the naturalness-based mutant selection performs similar (slightly worse) to the random mutant selection. Conclusions: Our findings are negative but we consider them interesting as they confute a strong intuition, i.e., fault revelation is independent of the mutants' naturalness.
Disciplines :
Computer science
Author, co-author :
Jimenez, Matthieu  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Titcheu Chekam, Thierry ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Cordy, Maxime  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Papadakis, Mike ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > Computer Science and Communications Research Unit (CSC)
Kintis, Marinos ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Le Traon, Yves ;  University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)
Harman, Mark;  University College London - UCL
External co-authors :
yes
Language :
English
Title :
Are mutants really natural? A study on how “naturalness” helps mutant selection
Publication date :
11 October 2018
Event name :
12th International Symposium on 
 Empirical Software Engineering and Measurement (ESEM'18)
Event place :
Oulu, Finland
Event date :
from 11-10-18 to 13-10-18
Audience :
International
Journal title :
Proceedings of 12th International Symposium on 
 Empirical Software Engineering and Measurement (ESEM'18)
Peer reviewed :
Peer reviewed
Focus Area :
Security, Reliability and Trust
Available on ORBilu :
since 06 October 2018

Statistics


Number of views
184 (25 by Unilu)
Number of downloads
403 (22 by Unilu)

Scopus citations®
 
13
Scopus citations®
without self-citations
12
OpenCitations
 
6
WoS citations
 
15

Bibliography


Similar publications



Contact ORBilu