On the Maintenance of System User Interactive Tests

Many companies rely on software testing to verify that their software products meet their requirements. As such, software testing is a keystone of the quality process. It offers means to assess that both functional and technical requirements are met. However, scaling the number of tests while preserving their quality poses challenges, just as in any software artifact. This is particularly true in the case of complex tests interacting with the SUT through its Graphical User Interface (GUI), i.e. System User Interactive Test (SUIT). The problem becomes challenging because as the System Under Test (SUT) evolves, the SUIT suites need to adapt and conform to the evolved software. Indeed, because the user interface is a part of the system that tends to undergo rapid evolutions, SUITs are particularly prompt to break.

As the first contribution of this dissertation, we aim to demonstrate the problem of test maintenance and overall improve the understanding of SUIT scripts evolution. To that end, we identify, collect and analyze test code changes across the evolution of an industrial test suite. We show that the problem of test maintenance is largely due to test fragility (most commonly performed changes are due to locator and synchronization issues) and bad practices (over 30% of keywords are duplicated).

To further investigate the question of bad test code practices such as test clones, we perform a multivocal study to identify which bad practices, i.e. SUIT smells, are already studied in both industry and academia. This process yields a catalog of 35 test code smells. For 16 of them, we derive metrics to analyze their diffusion across tests as well as potential refactoring actions removing the test code smells. Through an empirical study including both industrial and open-source test suites, we show that test code smells are largely present in SUITs, potentially contributing to SUIT fragility and hindering the maintenance process. Interestingly, refactoring actions remain rare during the lifespan of the tests. Yet, symptoms tend to disappear as old tests are discarded and new ones are introduced.

However, during the analysis of SUIT smells, we observe that bad practices impacting locators do not appear often in the test code. This observation contrasts with the analysis of the evolution of the SUIT presented in our first empirical study. This apparent contradiction arises from the limitation of property-based locators which rely on the structure of the representation layer of the SUT. Indeed, such approaches are sensitive to internal iso-functional changes occurring during the normal evolution of the SUT. To account for this limitation, we introduce a new way of expressing locators, HPath. Instead of relying on the internal representation of the presentation layer, HPath relies on the rendered characteristics of the GUI. Our results suggest that despite what is presented in the literature on locators, expressions relying on a smaller number of GUI elements to fully qualify a target do not necessarily lead to more robust locators. On the contrary, the choice of the GUI element properties seems to play a stronger role in the robustness to change.

Overall, this dissertation provides insights into how SUITs evolve and shows that SUIT fragility plays a major role in the associated maintenance effort. It also proposes techniques that effectively facilitate SUIT maintenance with the early detection of sub-optimal patterns and the introduction of a locator representation more robust to SUT evolution.