Multi-objective regression test selection in practice: an empirical study in the defense software industry

Executing an entire regression test suite after every code change is often costly in large software projects. To address this challenge, various regression test selection techniques have been proposed in the literature. One of those approaches is the Multi-Objective Regression Test Optimization (MORTO) approach, which is applied when there are multiple objectives during regression testing (e.g., minimizing the number of test cases and maximizing test coverage). This paper reports an action research type of study which was conducted to improve regression test-selection practices of a safety-critical industrial software in the defence domain based on the MORTO approach. The problem is formulated and solved by converting the multi-objective genetic-algorithm (GA) problem into a custom-built scalarized single-objective GA. The empirical results demonstrate that this approach yields a more efficient test suite (in terms of testing cost and benefits) compared to the old (manual) test-selection approach and another approach from the literature, i.e., the selective requirement coverage-based approach. Based on the context and need of the company, we developed the GA-based solution on a set of five cost objectives and four benefit objectives for regression testing while providing full coverage of the affected (changed) requirements. Since our proposed approach has been beneficial in the industrial context, it is currently in active use in the company.