[en] Replication is a fundamental pillar in the construction of scientific knowledge. Test data generation for procedural programs can be tackled using a single-target or a many-objective approach. The proponents of LIPS, a novel single-target test generator, conducted a preliminary empirical study to compare their approach with MOSA, an alternative many-objective test generator. However, their empirical investigation suffers from several external and internal validity threats, does not consider complex programs with many branches and does not include any qualitative analysis to interpret the results. In this paper, we report the results of a replication of the original study designed to address its major limitations and threats to validity. The new findings draw a completely different picture on the pros and cons of single-target vs many-objective approaches to test case generation.
Centre de recherche :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Software Verification and Validation Lab (SVV Lab)
Disciplines :
Sciences informatiques
Auteur, co-auteur :
PANICHELLA, Annibale ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Kifetew, Fitsum; Fondazione Bruno Kessler
Tonella, Paolo; Fondazione Bruno Kessler
Co-auteurs externes :
yes
Langue du document :
Anglais
Titre :
LIPS vs MOSA: a Replicated Empirical Study on Automated Test Case Generation
Date de publication/diffusion :
09 septembre 2017
Nom de la manifestation :
International Symposium on Search Based Software Engineering (SSBSE) 2017
Lieu de la manifestation :
Paderborn, Allemagne
Date de la manifestation :
from 09-09-2017 to 11-09-2017
Manifestation à portée :
International
Titre de l'ouvrage principal :
International Symposium on Search Based Software Engineering (SSBSE) 2017
Baker, R.D.: Modern permutation test software. In: Edgington, E. (ed.) Randomization Tests. Marcel Decker, New York (1995)
Conover, W.J.: Practical Nonparametric Statistics, 3rd edn. Wiley, New York (1998)
Deb, K., Deb, D.: Analysing mutation schemes for real-parameter genetic algorithms. Int. J. Artif. Intell. Soft Comput. 4(1), 1–28 (2014)
Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast elitist multi-objective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6, 182–197 (2000)
Fraser, G., Arcuri, A.: Whole test suite generation. IEEE Trans. Softw. Eng. 39(2), 276–291 (2013)
Fraser, G., Arcuri, A.: A large-scale evaluation of automated unit test generation using EvoSuite. ACM Trans. Softw. Eng. Methodol. 24(2), 8:1–8:42 (2014). http://doi.acm.org/10.1145/2685612
Juzgado, N.J., Vegas, S.: The role of non-exact replications in software engineering experiments. Empir. Softw. Eng. 16(3), 295–324 (2011)
McMinn, P.: Search-based software test data generation: a survey. Softw. Test. Verif. Reliab. 14(2), 105–156 (2004)
Panichella, A., Kifetew, F., Tonella, P.: Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets. IEEE Trans. Softw. Eng. PP(99), 1 (2017). Pre-print available online
Panichella, A., Kifetew, F.M., Tonella, P.: Reformulating branch coverage as a many-objective optimization problem. In: 8th IEEE International Conference on Software Testing, Verification and Validation, ICST, pp. 1–10 (2015)
Scalabrino, S., Grano, G., Nucci, D., Oliveto, R., Lucia, A.: Search-based testing of procedural programs: iterative single-target or multi-target approach? In: Sarro, F., Deb, K. (eds.) SSBSE 2016. LNCS, vol. 9962, pp. 64–79. Springer, Cham (2016). doi:10.1007/978-3-319-47106-8 5
Shull, F., Basili, V.R., Carver, J., Maldonado, J.C., Travassos, G.H., Mendonça, M.G., Fabbri, S.: Replicating software engineering experiments: addressing the tacit knowledge problem. In: 2002 International Symposium on Empirical Software Engineering (ISESE 2002), 3–4 October 2002, Nara, pp. 7–16 (2002)
Shull, F., Carver, J.C., Vegas, S., Juzgado, N.J.: The role of replications in empirical software engineering. Empir. Softw. Eng. 13(2), 211–218 (2008)
Tonella, P.: Evolutionary testing of classes. In: ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2004), pp. 119–128. ACM (2004)
Vargha, A., Delaney, H.D.: A critique and improvement of the CL common language effect size statistics of Mcgraw and Wong. J. Educ. Behav. Stat. 25(2), 101–132 (2000)
