Article (Scientific journals)
A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering
Arcuri, Andrea; Briand, Lionel
2012In Software Testing, Verification and Reliability
Peer reviewed
 

Files


Full Text
paper_stvr_icse_2012.pdf
Author preprint (659.45 kB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
statistical difference; effect size; parametric test; nonparametric test; confidence interval; Bonferroni adjustment; systematic review; survey
Abstract :
[en] Randomized algorithms are widely used to address many types of software engineering problems, especially in the area of software verification and validation with a strong emphasis on test automation. However, randomized algorithms are affected by chance and so require the use of appropriate statistical tests to be properly analysed in a sound manner. This paper features a systematic review regarding recent publications in 2009 and 2010 showing that, overall, empirical analyses involving randomized algorithms in software engineering tend to not properly account for the random nature of these algorithms. Many of the novel techniques presented clearly appear promising, but the lack of soundness in their empirical evaluations casts unfortunate doubts on their actual usefulness. In software engineering, although there are guidelines on how to carry out empirical analyses involving human subjects, those guidelines are not directly and fully applicable to randomized algorithms. Furthermore, many of the textbooks on statistical analysis are written from the viewpoints of social and natural sciences, which present different challenges from randomized algorithms. To address the questionable overall quality of the empirical analyses reported in the systematic review, this paper provides guidelines on how to carry out and properly analyse randomized algorithms applied to solve software engineering tasks, with a particular focus on software testing, which is by far the most frequent application area of randomized algorithms within software engineering.
Disciplines :
Computer science
Identifiers :
UNILU:UL-ARTICLE-2012-1227
Author, co-author :
Arcuri, Andrea;  Simula Research Laboratory, Norway
Briand, Lionel ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) ; University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)
External co-authors :
yes
Language :
English
Title :
A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering
Publication date :
2012
Journal title :
Software Testing, Verification and Reliability
ISSN :
0960-0833
Publisher :
John Wiley & Sons, Inc. - Engineering
Peer reviewed :
Peer reviewed
Available on ORBilu :
since 06 May 2013

Statistics


Number of views
237 (19 by Unilu)
Number of downloads
2384 (34 by Unilu)

Scopus citations®
 
401
Scopus citations®
without self-citations
352
OpenCitations
 
316
WoS citations
 
355

Bibliography


Similar publications



Contact ORBilu