An Empirical Study of Web Flaky Tests: Understanding and Unveiling DOM Event Interaction Challenges

PEI, Yu; PAPADAKIS, Mike; Jeongju Sohn

doi:10.1109/icst62969.2025.10989030

Download

Article (Scientific journals)

An Empirical Study of Web Flaky Tests: Understanding and Unveiling DOM Event Interaction Challenges

PEI, Yu; PAPADAKIS, Mike; Jeongju Sohn

2025 • In 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), p. 92-102

Peer reviewed

Permalink
https://hdl.handle.net/10993/65305

DOI
10.1109/icst62969.2025.10989030

Files (1)Send to Details Statistics Bibliography Similar publications

Files

Full Text

ICST25.pdf

Author postprint (447.96 kB)

Download

All documents in ORBilu are protected by a user license.

Send to

RIS BibTex APA Chicago Permalink X Linkedin

Details

Keywords :

Flaky tests, Web UI test, DOM and Event interaction

Disciplines :

Computer science

Author, co-author :

PEI, Yu ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal

PAPADAKIS, Mike ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SerVal

Jeongju Sohn; Kyungpook National University

External co-authors :

yes

Language :

English

Title :

An Empirical Study of Web Flaky Tests: Understanding and Unveiling DOM Event Interaction Challenges

Publication date :

20 May 2025

Journal title :

2025 IEEE Conference on Software Testing, Verification and Validation (ICST)

ISSN :

2159-4848

Publisher :

IEEE

Pages :

92-102

Peer reviewed :

Peer reviewed

Funders :

FNR - Luxembourg National Research Fund

Funding number :

C20/IS/14761415/TestFlakes

Available on ORBilu :

since 03 July 2025

Statistics

Number of views

64 (5 by Unilu)

Number of downloads

131 (3 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenCitations

OpenAlex citations

Bibliography

Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, "An empirical analysis of flaky tests, " in Proceedings of the ACM SIGSOFT Symposium on the Foundations of Software Engineering, vol. 16-21-November-2014, nov 2014, pp. 643-653.
A. Romano, Z. Song, S. Grandhi, W. Yang, and W. Wang, "An empirical analysis of ui-based flaky tests, " in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 1585-1597.
W. Lam, S. Winter, A. Wei, T. Xie, D. Marinov, and J. Bell, "A largescale longitudinal study of flaky tests, " Proceedings of the ACM on Programming Languages, vol. 4, no. OOPSLA, pp. 1-29, 2020.
O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, "A survey of flaky tests, " ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 1, pp. 1-74, 2021.
S. Habchi, G. Haben, M. Papadakis, M. Cordy, and Y. Le Traon, "A qualitative study on the sources, impacts, and mitigation strategies of flaky tests, " in 2022 IEEE Conference on Software Testing, Verification and Validation (ICST). IEEE, 2022, pp. 244-255.
G. Haben, S. Habchi, J. Micco, M. Harman, M. Papadakis, M. Cordy, and Y. L. Traon, "The importance of accounting for execution failures when predicting test flakiness, " in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, Industry Showcase, ASE 2024, Sacramento, California, United States. IEEE/ACM, 2024.
M. Eck, M. Castelluccio, F. Palomba, and A. Bacchelli, "Understanding Flaky Tests: The Developer's Perspective, " arXiv, pp. 830-840, 2019.
M. Machalica, A. Samylkin, M. Porth, and S. Chandra, "Predictive test selection, " in 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 2019, pp. 91-100.
T. Zimmermann, N. Nagappan, P. J. Guo, and B. Murphy, "Characterizing and predicting which bugs get reopened, " in 2012 34th International Conference on Software Engineering (ICSE). IEEE, 2012, pp. 1074-1083.
A. Memon, Z. Gao, B. Nguyen, S. Dhanda, E. Nickell, R. Siemborski, and J. Micco, "Taming google-scale continuous testing, " in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP). IEEE, 2017, pp. 233-242.
W. Lam, K. Muslu, H. Sajnani, and S. Thummalapenta, "A study on the lifecycle of flaky tests, " Proceedings-International Conference on Software Engineering, pp. 1471-1482, 2020.
J. Lampel, S. Just, S. Apel, and A. Zeller, "When life gives you oranges: detecting and diagnosing intermittent job failures at mozilla, " in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 1381-1392.
W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, "IDFlakies: A framework for detecting and partially classifying flaky tests, " Proceedings-2019 IEEE 12th International Conference on Software Testing, Verification and Validation, ICST 2019, pp. 312-322, 2019.
S. Habchi, G. Haben, J. Sohn, A. Franci, M. Papadakis, M. Cordy, and Y. Le Traon, "What made this test flake? pinpointing classes responsible for test flakiness, " in 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 352-363.
F. P. V. Pontillo and F. Ferrucci, "Static test flakiness prediction: How far can we go?" in Empirical Software Engineering, 2022.
O. Parry, G. M. Kapfhammer, M. Hilton, and P. McMinn, "Empirically evaluating flaky test detection techniques combining test case rerunning and machine learning models, " Empirical Software Engineering, vol. 28, no. 3, p. 72, 2023.
A. Marchetto, F. Ricca, and P. Tonella, "An empirical validation of a web fault taxonomy and its usage for web testing, " Journal of Web Engineering, pp. 316-345, 2009.
C. Leong, A. Singh, M. Papadakis, Y. L. Traon, and J. Micco, "Assessing transition-based test selection algorithms at google, " in Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice, ICSE (SEIP) 2019, Montreal, QC, Canada, May 25-31, 2019. IEEE/ACM, 2019, pp. 101-110. [Online]. Available: https://doi. org/10. 1109/ICSE-SEIP. 2019. 00019
C. Sung, M. Kusano, N. Sinha, and C. Wang, "Static dom event dependency analysis for testing web applications, " in Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2016, pp. 447-459.
D. Olianas, M. Leotta, F. Ricca, and L. Villa, "Reducing flakiness in endto-end test suites: An experience report, " in International Conference on the Quality of Information and Communications Technology. Springer, 2021, pp. 3-17.
A. M. Memon, A comprehensive framework for testing graphical user interfaces. University of Pittsburgh, 2001.
J. Strecker and A. M. Memon, "Testing graphical user interfaces, " in Encyclopedia of Information Science and Technology, Second Edition. IGI Global, 2009, pp. 3739-3744.
Z. Yu, F. Fahid, T. Menzies, G. Rothermel, K. Patrick, and S. Cherian, "Terminator: Better automated ui test case prioritization, " in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2019, pp. 883-894.
A. Mesbah, A. Van Deursen, and D. Roest, "Invariant-based automatic testing of modern web applications, " IEEE Transactions on Software Engineering, vol. 38, no. 1, pp. 35-53, 2011.
K. Barbosa, R. Ferreira, G. Pinto, M. d'Amorim, and B. Miranda, "Test flakiness across programming languages, " IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2039-2052, 2022.
A. Heiskanen, "Robotic process automation in automated gui testing of web applications, " 2021.
H. M. El-Bakry, A. M. Riad, M. Abu-Elsoud, S. Mohamed, A. E. Hassan, M. S. Kandel, and N. Mastorakis, "Adaptive user interface for web applications, " in Recent Advances in Business Administration: Proceedings of the 4th WSEAS International Conference on Business Administration (ICBA'10), 2010, pp. 20-22.
A. Mesbah and A. Van Deursen, "Invariant-based automatic testing of ajax user interfaces, " in 2009 IEEE 31st International Conference on Software Engineering. IEEE, 2009, pp. 210-220.
V. Garousi, A. Mesbah, A. Betin-Can, and S. Mirshokraie, "A systematic mapping study of web application testing, " Information and Software Technology, vol. 55, no. 8, pp. 1374-1396, 2013.
S. Dogan, A. Betin-Can, and V. Garousi, "Web application testing: A systematic literature review, " Journal of Systems and Software, vol. 91, pp. 174-201, 2014.
A. Milani Fard, M. Mirzaaghaei, and A. Mesbah, "Leveraging existing tests in automated test generation for web applications, " in Proceedings of the 29th ACM/IEEE international conference on Automated software engineering, 2014, pp. 67-78.
P. Aho and T. Vos, "Challenges in automated testing through graphical user interface, " in 2018 ieee international conference on software testing, verification and validation workshops (icstw). IEEE, 2018, pp. 118-121.
B. A. Myers and M. B. Rosson, "Survey on user interface programming, " in Proceedings of the SIGCHI conference on Human factors in computing systems, 1992, pp. 195-202.
Y. Pei, J. Sohn, S. Habchi, and M. Papadakis, "Non-flaky and nearlyoptimal time-based treatment of asynchronous wait web tests, " ACM Transactions on Software Engineering and Methodology, 2024.
C. Huo and J. Clause, "Improving oracle quality by detecting brittle assertions and unused inputs in tests, " in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, 2014, pp. 621-631.
N. Hashemi, A. Tahir, and S. Rasheed, "An empirical study of flaky tests in javascript, " in 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 24-34.
Z. Dong, A. Tiwari, X. L. Yu, and A. Roychoudhury, "Flaky test detection in android via event order exploration, " in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2021, pp. 367-378.
A. Wei, P. Yi, Z. Li, T. Xie, D. Marinov, and W. Lam, "Preempting flaky tests via non-idempotent-outcome tests, " in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 1730-1742.
J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, "Deflaker: Automatically detecting flaky tests, " in Proceedings of the 40th international conference on software engineering, 2018, pp. 433-444.
W. Lam, P. Godefroid, S. Nath, A. Santhiar, and S. Thummalapenta, "Root Causing Flaky Tests in a Large-Scale Industrial Setting, " in Proceedings ofthe 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA '19). Beijing, China: ACM Press, 2019, pp. 101-111.