Paper published in a book (Scientific congresses, symposiums and conference proceedings)
You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems
Liu, Kui; Koyuncu, Anil; Bissyande, Tegawendé François D Assise et al.
2019In The 12th IEEE International Conference on Software Testing, Verification and Validation (ICST-2019)
Peer reviewed
 

Files


Full Text
icst-2019.pdf
Publisher postprint (926.04 kB)
Download

All documents in ORBilu are protected by a user license.

Send to



Details



Keywords :
Automated Program Repair; Spectrum-based Fault Localization; Benchmarking; Empirical Assessment; Bias
Abstract :
[en] Properly benchmarking Automated Program Repair (APR) systems should contribute to the development and adoption of the research outputs by practitioners. To that end, the research community must ensure that it reaches significant milestones by reliably comparing state-of-the-art tools for a better understanding of their strengths and weaknesses. In this work, we identify and investigate a practical bias caused by the fault localization (FL) step in a repair pipeline. We propose to highlight the different fault localization configurations used in the literature, and their impact on APR systems when applied to the Defects4J benchmark. Then, we explore the performance variations that can be achieved by "tweaking'' the FL step. Eventually, we expect to create a new momentum for (1) full disclosure of APR experimental procedures with respect to FL, (2) realistic expectations of repairing bugs in Defects4J, as well as (3) reliable performance comparison among the state-of-the-art APR systems, and against the baseline performance results of our thoroughly assessed kPAR repair tool. Our main findings include: (a) only a subset of Defects4J bugs can be currently localized by commonly-used FL techniques; (b) current practice of comparing state-of-the-art APR systems (i.e., counting the number of fixed bugs) is potentially misleading due to the bias of FL configurations; and (c) APR authors do not properly qualify their performance achievement with respect to the different tuning parameters implemented in APR systems.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Security Design and Validation Research Group (SerVal)
Disciplines :
Computer science
Author, co-author :
Liu, Kui ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Koyuncu, Anil ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Bissyande, Tegawendé François D Assise  ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT)
Kim, Dongsun
Klein, Jacques ;  University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > Computer Science and Communications Research Unit (CSC)
Le Traon, Yves ;  University of Luxembourg > Faculty of Science, Technology and Communication (FSTC) > Computer Science and Communications Research Unit (CSC)
External co-authors :
no
Language :
English
Title :
You Cannot Fix What You Cannot Find! An Investigation of Fault Localization Bias in Benchmarking Automated Program Repair Systems
Publication date :
24 April 2019
Event name :
The 12th IEEE International Conference on Software Testing, Verification and Validation
Event place :
Xi'an, China
Event date :
from 22-04-2019 to 27-04-2019
Main work title :
The 12th IEEE International Conference on Software Testing, Verification and Validation (ICST-2019)
Publisher :
IEEE, Xi'an, China
Peer reviewed :
Peer reviewed
Focus Area :
Computational Sciences
FnR Project :
FNR10449467 - Automatic Bug Fix Recommendation: Improving Software Repair And Reducing Time-to-fix Delays In Software Development Projects, 2015 (01/02/2016-31/01/2019) - Tegawendé François D'assise Bissyandé
Funders :
FNR - Fonds National de la Recherche [LU]
Available on ORBilu :
since 21 January 2019

Statistics


Number of views
227 (19 by Unilu)
Number of downloads
131 (9 by Unilu)

Scopus citations®
 
88
Scopus citations®
without self-citations
70

Bibliography


Similar publications



Contact ORBilu