Azeem Ahmad, Ola Leifler, and Kristian Sandahl. 2019. Empirical Analysis of Factors and their Effect on Test Flakiness-Practitioners' Perceptions. CoRR abs/1906. 00673 (2019). arXiv:1906. 00673
Nadia Alshahwan, Andrea Ciancone, Mark Harman, Yue Jia, Ke Mao, Alexandru Marginean, Alexander Mols, Hila Peleg, Federica Sarro, and Ilya Zorin. 2019. Some Challenges for Software Testing Research (Invited Talk Paper). In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (Beijing, China) (ISSTA 2019). Association for Computing Machinery, New York, NY, USA, 1-3. https://doi. org/10. 1145/3293882. 3338991
Abdulrahman Alshammari, Christopher Morris, Michael Hilton, and Jonathan Bell. 2021. FlakeFlagger: Predicting Flakiness Without Rerunning Tests. In Proceedings of the IEEE/ACM 43rd International Conference on Software Engineering. IEEE, 1572-1584. https://doi. org/10. 1109/ICSE43902. 2021. 00140
Jonathan Bell, Owolabi Legunsen, Michael Hilton, Lamyaa Eloussi, Tifany Yung, and Darko Marinov. 2018. DeFlaker: Automatically Detecting Flaky Tests. In Proceedings of the 40th International Conference on Software Engineering-ICSE '18. ACM, 433-444. https://doi. org/10. 1145/3180155. 3180164
B. H. P. Camara, M. A. G. Silva, A. T. Endo, and S. R. Vergilio. 2021. What is the Vocabulary of Flaky Tests? An Extended Replication. In Proceedings of the 29th IEEE/ACM International Conference on Program Comprehension. IEEE/ACM, 11.
Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016). Association for Computing Machinery, New York, NY, USA, 449-452. https://doi. org/10. 1145/2931037. 2948707
Thomas Durieux, Fernanda Madeiral, Matias Martinez, and Rui Abreu. 2019. Empirical Review of Java Program Repair Tools: A Large-Scale Experiment on 2, 141 Bugs and 23, 551 Repair Attempts. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (Tallinn, Estonia) (ESEC/FSE 2019). Association for Computing Machinery, New York, NY, USA, 302-313. https://doi. org/10. 1145/3338906. 3338911
Saikat Dutta, August Shi, Rutvik Choudhary, Zhekun Zhang, Aryaman Jain, and Sasa Misailovic. 2020. Detecting flaky tests in probabilistic and machine learning applications. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 211-224. https://doi. org/10. 1145/3395363. 3397366
Moritz Eck, Fabio Palomba, Marco Castelluccio, and Alberto Bacchelli. 2019. Understanding Flaky Tests: The Developer's Perspective. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, 830-840. https: //doi. org/10. 1145/3338906. 3338945 arXiv:1907. 01466
Zebao Gao and Atif M. Memon. 2015. Which of My Failures are Real? Using Relevance Ranking to Raise True Failures to the Top. In Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering Workshop (ASEW). IEEE, 62-69. https://doi. org/10. 1109/ASEW. 2015. 7
Ali Ghanbari, Samuel Benton, and Lingming Zhang. 2019. Practical Program Repair via Bytecode Mutation. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (Beijing, China) (ISSTA 2019). Association for Computing Machinery, New York, NY, USA, 19-30. https://doi. org/10. 1145/3293882. 3330559
Martin Gruber, Stephan Lukasczyk, Florian Kroiß, and Gordon Fraser. 2021. An Empirical Study of Flaky Tests in Python. In Proceedings of the IEEE International Conference on Software Testing, Verification and Validation. IEEE, 11. arXiv:2101. 09077
Guillaume Haben, Sarra Habchi, Mike Papadakis, Maxime Cordy, and Yves Le Traon. 2021. A Replication Study on the Usability of Code Vocabulary in Predicting Flaky Tests. In Proceedings of the 18th International Conference on Mining Software Repositories. ACM, 11.
Mark Harman and Peter O'Hearn. 2018. From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis (keynote paper). In Proceedings of the 18th IEEE International Working Conference on Source Code Analysis and Manipulation. 1-23.
Facebook Inc. 2019. Facebook Testing and Verification request for proposals.
René Just, Darioush Jalali, and Michael D. Ernst. 2014. Defects4J: A Database of Existing Faults to Enable Controlled Testing Studies for Java Programs. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (San Jose, CA, USA) (ISSTA 2014). Association for Computing Machinery, New York, NY, USA, 437-440. https://doi. org/10. 1145/2610384. 2628055
Tariq M. King, Dionny Santiago, Justin Phillips, and Peter J. Clarke. 2018. Towards a Bayesian Network Model for Predicting Flaky Automated Tests. In Proceedings of the IEEE International Conference on Software Quality, Reliability and Security Companion. IEEE, 100-107. https://doi. org/10. 1109/QRS-C. 2018. 00031
Emily Kowalczyk, Karan Nair, Zebao Gao, Leo Silberstein, Teng Long, and Atif Memon. 2020. Modeling and ranking flaky tests at Apple. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Practice. ACM, 110-119. https://doi. org/10. 1145/3377813. 3381370
Adriaan Labuschagne, Laura Inozemtseva, and Reid Holmes. 2017. Measuring the cost of regression testing in practice: a study of Java projects using continuous integration. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering. ACM, 821-830. https://doi. org/10. 1145/3106237. 3106288
Wing Lam, Patrice Godefroid, Suman Nath, Anirudh Santhiar, and Suresh Thummalapenta. 2019. Root causing flaky tests in a large-scale industrial setting. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, Beijing, China, 101-111. https://doi. org/10. 1145/3293882. 3330570
Wing Lam, Kivanç Muslu, Hitesh Sajnani, and Suresh Thummalapenta. 2020. A study on the lifecycle of flaky tests. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. ACM, 1471-1482. https://doi. org/10. 1145/3377811. 3381749
Wing Lam, Reed Oei, August Shi, Darko Marinov, and Tao Xie. 2019. iDFlakies: A Framework for Detecting and Partially Classifying Flaky Tests. In Proceedings of the 12th IEEE Conference on Software Testing, Validation and Verification (ICST). IEEE, 312-322. https://doi. org/10. 1109/ICST. 2019. 00038
Wing Lam, August Shi, Reed Oei, Sai Zhang, Michael D. Ernst, and Tao Xie. 2020. Dependent-test-aware regression testing techniques. In Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 298-311. https://doi. org/10. 1145/3395363. 3397364
Johannes Lampel, Sascha Just, Sven Apel, and Andreas Zeller. 2021. When life gives you oranges: detecting and diagnosing intermittent job failures at Mozilla. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. ACM, New York, NY, USA, 1381-1392. https://doi. org/10. 1145/3468264. 3473931
Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, andWestleyWeimer. 2012. GenProg: A Generic Method for Automatic Software Repair. IEEE Transactions on Software Engineering 38, 1 (2012), 54-72. https://doi. org/10. 1109/TSE. 2011. 104
Claire Leong, Abhayendra Singh, Mike Papadakis, Yves Le Traon, and John Micco. 2019. Assessing Transition-Based Test Selection Algorithms at Google. In Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice (Montreal, Quebec, Canada) (ICSE-SEIP '19). IEEE Press, 101-110. https://doi. org/10. 1109/ICSE-SEIP. 2019. 00019
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In 22nd International Symposium on Foundations of Software Engineering (FSE 2014), Shing-Chi Cheung, Alessandro Orso, and Margaret-Anne Storey (Eds.). ACM, Hong Kong, China, 643-653.
Jean Malm, Adnan Causevic, Björn Lisper, and Sigrid Eldh. 2020. Automated Analysis of Flakiness-mitigating Delays. In Proceedings of the IEEE/ACM 1st International Conference on Automation of Software Test. ACM, 81-84. https: //doi. org/10. 1145/3387903. 3389320
Atif M. Memon and Myra B. Cohen. 2013. Automated testing of GUI applications: models, tools, and controlling flakiness. In Proceedings of the 35th International Conference on Software Engineering, David Notkin, Betty H. C. Cheng, and Klaus Pohl (Eds.). IEEE Computer Society, San Francisco, CA, USA, 1479-1480.
Atif M. Memon, Zebao Gao, Bao N. Nguyen, Sanjeev Dhanda, Eric Nickell, Rob Siemborski, and John Micco. 2017. Taming Google-Scale Continuous Testing. In 39th International Conference on Software Engineering, Software Engineering in Practice Track (ICSE-SEIP). IEEE, Buenos Aires, Argentina, 233-242.
Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six-Mutation Testing Advances: An Analysis and Survey. In Advances in Computers, Atif M. Memon (Ed.). Advances in Computers, Vol. 112. Elsevier, 275-378. https://doi. org/10. 1016/bs. adcom. 2018. 03. 015
Owain Parry, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. 2020. Flake It 'Till You Make It. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops. ACM, 11-12. https://doi. org/10. 1145/3387940. 3392177
Samad Paydar and Aidin Azamnouri. 2019. An Experimental Study on Flakiness and Fragility of Randoop Regression Test Suites. In Fundamentals of Software Engineering, Hossein Hojjat and Mieke Massink (Eds.). Springer International Publishing, 111-126. https://doi. org/10. 1007/978-3-030-31517-7_8
Gustavo Pinto, Breno Miranda, Supun Dissanayake, Marcelo D'Amorim, Christoph Treude, and Antonia Bertolino. 2020. What is the Vocabulary of Flaky Tests?. In Proceedings of the 17th International Conference on Mining Software Repositories. ACM, 492-502. https://doi. org/10. 1145/3379597. 3387482
Kai Presler-Marshall, Eric Horton, Sarah Heckman, and Kathryn T Stolee. 2019. Wait Wait. No, Tell Me. Analyzing Selenium Configuration Effects on Test Flakiness. In Proceedings of the IEEE/ACM 14th International Workshop on Automation of Software Test (AST). IEEE, Montreal, Canada, 2-8. https: //doi. org/10. 1109/AST. 2019. 000-1
Yihao Qin, Shangwen Wang, Kui Liu, Xiaoguang Mao, and Tegawende F. Bissyande. 2021. On the Impact of Flaky Tests in Automated Program Repair. In Proceedings of the IEEE International Conference on Software Analysis, Evolution and Reengineering. IEEE, 295-306. https://doi. org/10. 1109/SANER50967. 2021. 00035
August Shi, Jonathan Bell, and Darko Marinov. 2019. Mitigating the effects of flaky tests on mutation testing. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. ACM, 112-122. https: //doi. org/10. 1145/3293882. 3330568
August Shi, Wing Lam, Reed Oei, Tao Xie, and Darko Marinov. 2019. IFixFlakies: A Framework for Automatically Fixing Order-Dependent Flaky Tests. In Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2019). Association for Computing Machinery, 545-555. https://doi. org/10. 1145/ 3338906. 3338925
Denini Silva, Leopoldo Teixeira, and Marcelo D'Amorim. 2020. Shake It! Detecting Flaky Tests Caused by Concurrency with Shaker. In Proceedings of the IEEE International Conference on Software Maintenance and Evolution. IEEE, 301-311. https://doi. org/10. 1109/ICSME46990. 2020. 00037
Bela Vancsics, Tamas Gergely, and Arpad Beszedes. 2020. Simulating the Effect of Test Flakiness on Fault Localization Effectiveness. In Proceedings of the IEEE Workshop on Validation, Analysis and Evolution of Software Tests. IEEE, 28-35. https://doi. org/10. 1109/VST50071. 2020. 9051636
Roberto Verdecchia, Emilio Cruciani, Breno Miranda, and Antonia Bertolino. 2021. Know You Neighbor: Fast Static Prediction of Test Flakiness. IEEE Access 9 (2021), 76119-76134. https://doi. org/10. 1109/ACCESS. 2021. 3082424
W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A Survey on Software Fault Localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707-740. https://doi. org/10. 1109/TSE. 2016. 2521368
Yuan Yuan and Wolfgang Banzhaf. 2020. ARJA: Automated Repair of Java Programs via Multi-Objective Genetic Programming. IEEE Transactions on Software Engineering 46, 10 (2020), 1040-1067. https://doi. org/10. 1109/TSE. 2018. 2874648
