Comparing White-box and Black-box Test Prioritization

in 38th International Conference on Software Engineering (ICSE'16) (2016)

Although white-box regression test prioritization has been well-studied, the more recently introduced black-box prioritization approaches have neither been compared against each other nor against more ... [more ▼]

Although white-box regression test prioritization has been well-studied, the more recently introduced black-box prioritization approaches have neither been compared against each other nor against more well-established white-box techniques. We present a comprehensive experimental comparison of several test prioritization techniques, including well-established white-box strategies and more recently introduced black-box approaches. We found that Combinatorial Interaction Testing and diversity-based techniques (Input Model Diversity and Input Test Set Diameter) perform best among the black-box approaches. Perhaps surprisingly, we found little difference between black-box and white-box performance (at most 4% fault detection rate difference). We also found the overlap between black- and white-box faults to be high: the first 10% of the prioritized test suites already agree on at least 60% of the faults found. These are positive findings for practicing regression testers who may not have source code available, thereby making white-box techniques inapplicable. We also found evidence that both black-box and white-box prioritization remain robust over multiple system releases. [less ▲]

Flattening or not of the combinatorial interaction testing models

in Eighth IEEE International Conference on Software Testing, Verification and Validation, ICST 2015 Workshops (2015, April)

Similarity testing for access control

in Information and Software Technology (2015), 58

Context: Access control is among the most important security mechanisms, and XACML is the de facto standard for specifying, storing and deploying access control policies. Since it is critical that ... [more ▼]

Context: Access control is among the most important security mechanisms, and XACML is the de facto standard for specifying, storing and deploying access control policies. Since it is critical that enforced policies are correct, policy testing must be performed in an effective way to identify potential security flaws and bugs. In practice, exhaustive testing is impossible due to budget constraints. Therefore the tests need to be prioritized so that resources are focused on their most relevant subset. Objective: This paper tackles the issue of access control test prioritization. It proposes a new approach for access control test prioritization that relies on similarity. Method: The approach has been applied to several policies and the results have been compared to random prioritization (as a baseline). To assess the different prioritization criteria, we use mutation analysis and compute the mutation scores reached by each criterion. This helps assessing the rate of fault detection. Results: The empirical results indicate that our proposed approach is effective and its rate of fault detection is higher than that of random prioritization. Conclusion: We conclude that prioritization of access control test cases can be usefully based on similarity criteria. © 2014 Elsevier B.V. All rights reserved. [less ▲]

MutaLog: a Tool for Mutating Logic Formulas

in Testing Tools Track, 7th International Conference on Software Testing, Verification and Validation (ICST 2014) (2014)

Sampling Program Inputs with Mutation Analysis: Going Beyond Combinatorial Interaction Testing

in 7th International Conference on Software Testing, Verification and Validation (ICST 2014) (2014)

Towards a Language-Independent Approach for Reverse-Engineering of Software Product Lines

in 29th Symposium on Applied Computing (SAC 2014) (2014)

Towards automated testing and fixing of re-engineered feature models

in Proceedings of the 2013 International Conference on Software Engineering (2013)

Multi-objective test generation for software product lines

in 17th International Software Product Line Conference, SPLC 2013, Tokyo, Japan - August 26 - 30, 2013 (2013)