Reference : On the Impact of Flaky Tests in Automated Program Repair
Scientific congresses, symposiums and conference proceedings : Paper published in a book
Engineering, computing & technology : Computer science
Security, Reliability and Trust
On the Impact of Flaky Tests in Automated Program Repair
Qin, Yihao mailto [National University of Defense Technology, Changsha, China]
Wang, Shangwen mailto [National University of Defense Technology, Changsha, China]
Liu, Kui mailto [Nanjing University of Aeronautics and Astronautics, Nanjing, China]
Mao, Xiaoguang mailto [National University of Defense Technology, Changsha, China]
Bissyande, Tegawendé François D Assise mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > TruX >]
28th IEEE International Conference on Software Analysis, Evolution and Reengineering, Hawaii 9-12 March 2021
28th IEEE International Conference on Software Analysis, Evolution and Reengineering
9-03-2021 to 12-03-2021
United States
[en] Program repair ; Flaky tests ; Empirical assessment
[en] The literature of Automated Program Repair is largely dominated by approaches that leverage test suites not only to expose bugs but also to validate the generated patches. Unfortunately, beyond the widely-discussed concern that test suites are an imperfect oracle because they can be incomplete, they can include tests that are flaky. A flaky test is one that can be passed or failed by a program in a non-deterministic way. Such tests are generally carefully removed from the repair benchmarks. In practice, however, flaky tests are available test suite of software repositories. To the best of our knowledge, no study has discussed this threat to validity for evaluation of program repair. In this work, we highlight this threat and further investigate the impact of flaky tests by reverting their removal from the Defects4J benchmark. Our study aims to characterize the impact of flaky tests for localizing bugs and the eventual influence on the repair performance. Among other insights, we find that (1) although flaky tests are few (≈0.3%) of total tests, they affect experiments related to a large proportion (98.9%) of Defects4J real-world faults; (2) most flaky tests (98%) actually provide deterministic results under specific environment configurations (with the jdk version influencing the results); (3) flaky tests drastically hinder the effectiveness of spectrum-based fault localization (e.g., the rankings of 90 bugs drop down while none of the bugs obtains better location results compared with results achieved without flaky tests); and (4) the repairability of APR tools is greatly affected by the presence of flaky tests (e.g., 10 state of the art APR tools can now fix significantly fewer bugs than when the benchmark is manually curated to remove flaky tests). Given that the detection of flaky tests is still nascent, we call for the program repair community to relax the artificial assumption that the test suite is free from flaky tests. One direction that we propose is to consider developing strategies where patches that partially-fix bugs are considered worthwhile: a patch may make the program pass some test cases but fail some (which may actually be the flaky ones).
Researchers ; General public

File(s) associated to this reference

Fulltext file(s):

Open access
2021_SANER_Flakytests4APR.pdfAuthor postprint404.59 kBView/Open

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.