References of "Fraser, Gordon"
     in
Bookmark and Share    
Full Text
Peer Reviewed
See detailAn Empirical Evaluation of Evolutionary Algorithms for Unit Test Suite Generation
Campos, Jose; Ge, Yan; Albunian, Nasser et al

in Information and Software Technology (2018), 104(December), 207-235

Context: Evolutionary algorithms have been shown to be e ective at generating unit test suites optimised for code coverage. While many speci c aspects of these algorithms have been evaluated in detail (e ... [more ▼]

Context: Evolutionary algorithms have been shown to be e ective at generating unit test suites optimised for code coverage. While many speci c aspects of these algorithms have been evaluated in detail (e.g., test length and di erent kinds of techniques aimed at improving performance, like seeding), the in uence of the choice of evolutionary algorithm has to date seen less attention in the literature. Objective: Since it is theoretically impossible to design an algorithm that is the best on all possible problems, a common approach in software engineering problems is to rst try the most common algorithm, a Genetic Algorithm, and only afterwards try to re ne it or compare it with other algorithms to see if any of them is more suited for the addressed problem. The objective of this paper is to perform this analysis, in order to shed light on the in uence of the search algorithm applied for unit test generation. Method: We empirically evaluate thirteen di erent evolutionary algorithms and two random approaches on a selection of non-trivial open source classes. All algorithms are implemented in the EvoSuite test generation tool, which includes recent optimisations such as the use of an archive during the search and optimisation for multiple coverage criteria. Results: Our study shows that the use of a test archive makes evolutionary algorithms clearly better than random testing, and it con rms that the DynaMOSA many-objective search algorithm is the most e ective algorithm for unit test generation. Conclusions: Our results show that the choice of algorithm can have a substantial in uence on the performance of whole test suite optimisation. Although we can make a recommendation on which algorithm to use in practice, no algorithm is clearly superior in all cases, suggesting future work on improved search algorithms for unit test generation [less ▲]

Detailed reference viewed: 112 (51 UL)
Full Text
Peer Reviewed
See detailEvoSuite at the SBST 2018 Tool Competition
Fraser, Gordon; Rojas, Jose; Arcuri, Andrea UL

in 2018 ACM/IEEE 11th International Workshop on Search-Based Software Testing (2018)

EvoSuite is a search-based tool that automatically generates executable unit tests for Java code (JUnit tests). This paper summarises the results and experiences of EvoSuite’s participation at the sixth ... [more ▼]

EvoSuite is a search-based tool that automatically generates executable unit tests for Java code (JUnit tests). This paper summarises the results and experiences of EvoSuite’s participation at the sixth unit testing competition at SBST 2018, where EvoSuite achieved the highest overall score (687 points) for the fifth time in six editions of the competition. [less ▲]

Detailed reference viewed: 40 (3 UL)
Full Text
Peer Reviewed
See detailPrivate API Access and Functional Mocking in Automated Unit Test Generation
Arcuri, Andrea UL; Fraser, Gordon; Just, Rene

in IEEE International Conference on Software Testing, Verification and Validation (ICST) (2017)

Not all object oriented code is easily testable: Dependency objects might be difficult or even impossible to instantiate, and object-oriented encapsulation makes testing potentially simple code difficult ... [more ▼]

Not all object oriented code is easily testable: Dependency objects might be difficult or even impossible to instantiate, and object-oriented encapsulation makes testing potentially simple code difficult if it cannot easily be accessed. When this happens, then developers can resort to mock objects that simulate the complex dependencies, or circumvent objectoriented encapsulation and access private APIs directly through the use of, for example, Java reflection. Can automated unit test generation benefit from these techniques as well? In this paper we investigate this question by extending the EvoSuite unit test generation tool with the ability to directly access private APIs and to create mock objects using the popular Mockito framework. However, care needs to be taken that this does not impact the usefulness of the generated tests: For example, a test accessing a private field could later fail if that field is renamed, even if that renaming is part of a semantics-preserving refactoring. Such a failure would not be revealing a true regression bug, but is a false positive, which wastes the developer’s time for investigating and fixing the test. Our experiments on the SF110 and Defects4J benchmarks confirm the anticipated improvements in terms of code coverage and bug finding, but also confirm the existence of false positives. However, by ensuring the test generator only uses mocking and reflection if there is no other way to reach some part of the code, their number remains small. [less ▲]

Detailed reference viewed: 93 (9 UL)
Full Text
Peer Reviewed
See detailEVOSUITE at the SBST 2017 Tool Competition
Fraser, Gordon; Rojas, José Miguel; Campos, José et al

in IEEE/ACM International Workshop on Search-Based Software Testing (SBST) (2017)

EVOSUITE is a search-based tool that automatically generates unit tests for Java code. This paper summarises the results and experiences of EVOSUITE’s participation at the fifth unit testing competition ... [more ▼]

EVOSUITE is a search-based tool that automatically generates unit tests for Java code. This paper summarises the results and experiences of EVOSUITE’s participation at the fifth unit testing competition at SBST 2017, where EVOSUITE achieved the highest overall score. [less ▲]

Detailed reference viewed: 74 (7 UL)
Full Text
Peer Reviewed
See detailAn Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application
Almasi, Moein; Hemmati, Hadi; Fraser, Gordon et al

in ACM/IEEE International Conference on Software Engineering (ICSE) (2017)

Automated unit test generation has been extensively studied in the literature in recent years. Previous studies on open source systems have shown that test generation tools are quite effective at ... [more ▼]

Automated unit test generation has been extensively studied in the literature in recent years. Previous studies on open source systems have shown that test generation tools are quite effective at detecting faults, but how effective and applicable are they in an industrial application? In this paper, we investigate this question using a life insurance and pension products calculator engine owned by SEB Life & Pension Holding AB Riga Branch. To study fault-finding effectiveness, we extracted 25 real faults from the version history of this software project, and applied two up-to-date unit test generation tools for Java, EvoSuite and Randoop, which implement search-based and feedback-directed random test generation, respectively. Automatically generated test suites detected up to 56.40% (EvoSuite) and 38.00% (Randoop) of these faults. The analysis of our results demonstrates challenges that need to be addressed in order to improve fault detection in test generation tools. In particular, classification of the undetected faults shows that 97.62% of them depend on either “specific primitive values” (50.00%) or the construction of “complex state configuration of objects” (47.62%). To study applicability, we surveyed the developers of the application under test on their experience and opinions about the test generation tools and the generated test cases. This leads to insights on requirements for academic prototypes for successful technology transfer from academic research to industrial practice, such as a need to integrate with popular build tools, and to improve the readability of the generated tests. [less ▲]

Detailed reference viewed: 304 (13 UL)
Full Text
Peer Reviewed
See detailAn Empirical Evaluation of Evolutionary Algorithms for Test Suite Generation
Campos, Jose; Ge, Yan; Fraser, Gordon et al

in Symposium on Search-Based Software Engineering (SSBSE) (2017)

Evolutionary algorithms have been shown to be effective at generating unit test suites optimised for code coverage. While many aspects of these algorithms have been evaluated in detail (e.g., test length ... [more ▼]

Evolutionary algorithms have been shown to be effective at generating unit test suites optimised for code coverage. While many aspects of these algorithms have been evaluated in detail (e.g., test length and different kinds of techniques aimed at improving performance, like seeding), the influence of the specific algorithms has to date seen less attention in the literature. As it is theoretically impossible to design an algorithm that is best on all possible problems, a common approach in software engineering problems is to first try a Genetic Algorithm, and only afterwards try to refine it or compare it with other algorithms to see if any of them is more suited for the addressed problem. This is particularly important in test generation, since recent work suggests that random search may in practice be equally effective, whereas the reformulation as a many-objective problem seems to be more effective. To shed light on the influence of the search algorithms, we empirically evaluate six different algorithms on a selection of non-trivial open source classes. Our study shows that the use of a test archive makes evolutionary algorithms clearly better than random testing, and it confirms that the many-objective search is the most effective. [less ▲]

Detailed reference viewed: 144 (28 UL)
Full Text
Peer Reviewed
See detailSeeding Strategies in Search-Based Unit Test Generation
Rojas, Jose Miguel; Fraser, Gordon; Arcuri, Andrea UL

in SOFTWARE TESTING, VERIFICATION AND RELIABILITY (STVR) (2016)

Search-based techniques have been applied successfully to the task of generating unit tests for object-oriented software. However, as for any meta-heuristic search, the efficiency heavily depends on many ... [more ▼]

Search-based techniques have been applied successfully to the task of generating unit tests for object-oriented software. However, as for any meta-heuristic search, the efficiency heavily depends on many factors; seeding, which refers to the use of previous related knowledge to help solve the testing problem at hand, is one such factor that may strongly influence this efficiency. This paper investigates different seeding strategies for unit test generation, in particular seeding of numerical and string constants derived statically and dynamically, seeding of type information, and seeding of previously generated tests. To understand the effects of these seeding strategies, the results of a large empirical analysis carried out on a large collection of open source projects from the SF110 corpus and the Apache Commons repository are reported. These experiments show with strong statistical confidence that, even for a testing tool already able to achieve high coverage, the use of appropriate seeding strategies can further improve performance. [less ▲]

Detailed reference viewed: 122 (32 UL)
Full Text
Peer Reviewed
See detailA Detailed Investigation of the Effectiveness of Whole Test Suite Generation
Rojas, José Miguel; Vivanti, Mattia; Arcuri, Andrea UL et al

in Empirical Software Engineering (2016)

A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., lines, branches, mutants). Rather than generating one test case at a ... [more ▼]

A common application of search-based software testing is to generate test cases for all goals defined by a coverage criterion (e.g., lines, branches, mutants). Rather than generating one test case at a time for each of these goals individually, whole test suite generation optimizes entire test suites towards satisfying all goals at the same time. There is evidence that the overall coverage achieved with this approach is superior to that of targeting individual coverage goals. Nevertheless, there remains some uncertainty on (a) whether the results generalize beyond branch coverage, (b) whether the whole test suite approach might be inferior to a more focused search for some particular coverage goals, and (c) whether generating whole test suites could be optimized by only targeting coverage goals not already covered. In this paper, we perform an in-depth analysis to study these questions. An empirical study on 100 Java classes using three different coverage criteria reveals that indeed there are some testing goals that are only covered by the traditional approach, although their number is only very small in comparison with those which are exclusively covered by the whole test suite approach. We find that keeping an archive of already covered goal with corresponding tests and focusing the search on uncovered goals overcomes this small drawback on larger classes, leading to an improved overall effectiveness of whole test suite generation. [less ▲]

Detailed reference viewed: 117 (17 UL)
Full Text
Peer Reviewed
See detailJava Enterprise Edition Support in Search-Based JUnit Test Generation
Arcuri, Andrea UL; Fraser, Gordon

in Java Enterprise Edition Support in Search-Based JUnit Test Generation (2016)

Many di fferent techniques and tools for automated unit test generation target the Java programming languages due to its popular- ity. However, a lot of Java's popularity is due to its usage to develop ... [more ▼]

Many di fferent techniques and tools for automated unit test generation target the Java programming languages due to its popular- ity. However, a lot of Java's popularity is due to its usage to develop enterprise applications with frameworks such as Java Enterprise Edition (JEE) or Spring. These frameworks pose challenges to the automatic gen- eration of JUnit tests. In particular, code units (\beans") are handled by external web containers (e.g., WildFly and GlassFish). Without consid- ering how web containers initialize these beans, automatically generated unit tests would not represent valid scenarios and would be of little use. For example, common issues of bean initialization are dependency in- jection, database connection, and JNDI bean lookup. In this paper, we extend the EvoSuite search-based JUnit test generation tool to provide initial support for JEE applications. Experiments on 247 classes (the JBoss EAP tutorial examples) reveal an increase in code coverage, and demonstrate that our techniques prevent the generation of useless tests (e.g., tests where dependencies are not injected). [less ▲]

Detailed reference viewed: 102 (8 UL)
Full Text
Peer Reviewed
See detailEvoSuite at the SBST 2015 Tool Competition
Fraser, Gordon; Arcuri, Andrea UL

in 8th International Workshop on Search-Based Software Testing (SBST) 2015 (2015)

EVOSUITE is a mature research prototype that automatically generates unit tests for Java code. This paper summarizes the results and experiences of EVOSUITE’s participation at the third unit testing ... [more ▼]

EVOSUITE is a mature research prototype that automatically generates unit tests for Java code. This paper summarizes the results and experiences of EVOSUITE’s participation at the third unit testing competition at SBST 2015. An unfortunate issue of conflicting dependency versions in two out of the nine benchmark projects reduced EVOSUITE’s overall score to 190:6, leading to the overall second rank. [less ▲]

Detailed reference viewed: 61 (5 UL)
Full Text
Peer Reviewed
See detailDo Automatically Generated Unit Tests Find Real Faults? An Empirical Study of Effectiveness and Challenges
Shamshiri, Sina; Just, Rene; Rojas, Jose Miguel et al

in Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2015)

Rather than tediously writing unit tests manually, tools can be used to generate them automatically — sometimes even resulting in higher code coverage than manual testing. But how good are these tests at ... [more ▼]

Rather than tediously writing unit tests manually, tools can be used to generate them automatically — sometimes even resulting in higher code coverage than manual testing. But how good are these tests at actually finding faults? To answer this question, we applied three state-of-the art unit test generation tools for Java (Randoop, EvoSuite, and Agitar) to the 357 faults in the Defects4J dataset and investigated how well the generated test suites perform at detecting faults. Although 55.7% of the faults were found by automatically generated tests overall, only 19.9% of the test suites generated in our experiments actually detected a fault. By studying the performance and the problems of the individual tools and their tests, we derive insights to support the development of automated unit test generators, in order to increase the fault detection rate in the future. These include 1) improving coverage obtained so that defective statements are actually executed in the first instance, 2) techniques for propagating faults to the output, coupled with the generation of more sensitive assertions for detecting them, and 3) better simulation of the execution environment to detecting faults that are dependent on external factors, for example the date and time. [less ▲]

Detailed reference viewed: 132 (7 UL)
Full Text
Peer Reviewed
See detailCombining Multiple Coverage Criteria in Search-Based Unit Test Generation
Rojas, Miguel; Campos, Jose; Vivanti, Mattia et al

in Symposium on Search-Based Software Engineering (SSBSE) (2015)

Automated test generation techniques typically aim at max- imising coverage of well-established structural criteria such as statement or branch coverage. In practice, generating tests only for one speci c ... [more ▼]

Automated test generation techniques typically aim at max- imising coverage of well-established structural criteria such as statement or branch coverage. In practice, generating tests only for one speci c criterion may not be su cient when testing object oriented classes, as standard structural coverage criteria do not fully capture the properties developers may desire of their unit test suites. For example, covering a large number of statements could be easily achieved by just calling the main method of a class; yet, a good unit test suite would consist of smaller unit tests invoking individual methods, and checking return values and states with test assertions. There are several di erent properties that test suites should exhibit, and a search-based test generator could easily be extended with additional tness functions to capture these properties. However, does search-based testing scale to combinations of multiple cri- teria, and what is the e ect on the size and coverage of the resulting test suites? To answer these questions, we extended the EvoSuite unit test generation tool to support combinations of multiple test criteria, de ned and implemented several di erent criteria, and applied combinations of criteria to a sample of 650 open source Java classes. Our experiments suggest that optimising for several criteria at the same time is feasible without increasing computational costs: When combining nine di erent criteria, we observed an average decrease of only 0.4% for the constituent coverage criteria, while the test suites may grow up to 70%. [less ▲]

Detailed reference viewed: 83 (8 UL)
Full Text
Peer Reviewed
See detailGenerating TCP/UDP Network Data for Automated Unit Test Generation
Arcuri, Andrea UL; Fraser, Gordon; Galeotti, Juan Pablo

in 10th Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE) (2015)

Although automated unit test generation techniques can in principle generate test suites that achieve high code coverage, in practice this is often inhibited by the dependence of the code under test on ... [more ▼]

Although automated unit test generation techniques can in principle generate test suites that achieve high code coverage, in practice this is often inhibited by the dependence of the code under test on external resources. In particular, a common problem in modern programming languages is posed by code that involves networking (e.g., opening a TCP listening port). In order to generate tests for such code, we describe an approach where we mock (simulate) the networking interfaces of the Java standard library, such that a search-based test generator can treat the network as part of the test input space. This not only has the benefit that it overcomes many limitations of testing networking code (e.g., different tests binding to the same local ports, and deterministic resolution of hostnames and ephemeral ports), it also substantially increases code coverage. An evaluation on 23,886 classes from 110 open source projects, totalling more than 6.6 million lines of Java code, reveals that network access happens in 2,642 classes (11%). Our implementation of the proposed technique as part of the EVOSUITE testing tool addresses the networking code contained in 1,672 (63%) of these classes, and leads to an increase of the average line coverage from 29.1% to 50.8%. On a manual selection of 42 Java classes heavily depending on networking, line coverage with EVOSUITE more than doubled with the use of network mocking, increasing from 31.8% to 76.6%. [less ▲]

Detailed reference viewed: 98 (6 UL)
Full Text
Peer Reviewed
See detailContinuous Test Generation on Guava
Campos, Jose; Fraser, Gordon; Arcuri, Andrea UL et al

in Symposium on Search-Based Software Engineering (SSBSE) (2015)

Search-based testing can be applied to automatically gener- ate unit tests that achieve high levels of code coverage on object-oriented classes. However, test generation takes time, in particular if ... [more ▼]

Search-based testing can be applied to automatically gener- ate unit tests that achieve high levels of code coverage on object-oriented classes. However, test generation takes time, in particular if projects consist of many classes, like in the case of the Guava library. To allow search-based test generation to scale up and to integrate it better into software development, continuous test generation applies test generation incrementally during continuous integration. In this paper, we report on the application of continuous test generation with EvoSuite at the SS- BSE'15 challenge on the Guava library. Our results show that continuous test generation reduces the time spent on automated test generation by 96%, while increasing code coverage by 13.9% on average. [less ▲]

Detailed reference viewed: 53 (9 UL)
Full Text
Peer Reviewed
See detailAutomated Unit Test Generation during Software Development: A Controlled Experiment and Think-Aloud Observations
Rojas, José Miguel; Fraser, Gordon; Arcuri, Andrea UL

in ACM International Symposium on Software Testing and Analysis (ISSTA), 2015 (2015)

Automated unit test generation tools can produce tests that are superior to manually written ones in terms of code coverage, but are these tests helpful to developers while they are writing code? A ... [more ▼]

Automated unit test generation tools can produce tests that are superior to manually written ones in terms of code coverage, but are these tests helpful to developers while they are writing code? A developer would first need to know when and how to apply such a tool, and would then need to understand the resulting tests in order to provide test oracles and to diagnose and fix any faults that the tests reveal. Considering all this, does automatically generating unit tests provide any benefit over simply writing unit tests manually? We empirically investigated the effects of using an automated unit test generation tool (EVOSUITE) during development. A controlled experiment with 41 students shows that using EVOSUITE leads to an average branch coverage increase of +13%, and 36% less time is spent on testing compared to writing unit tests manually. However, there is no clear effect on the quality of the implementations, as it depends on how the test generation tool and the generated tests are used. In-depth analysis, using five think-aloud observations with professional programmers, confirms the necessity to increase the usability of automated unit test generation tools, to integrate them better during software development, and to educate software developers on how to best use those tools. [less ▲]

Detailed reference viewed: 89 (19 UL)
Full Text
Peer Reviewed
See detailDoes automated white-box test generation really help software testers?
Fraser, Gordon; Staats, Matthew UL; McMinn, Phil et al

in ISSTA (2013)

Detailed reference viewed: 49 (14 UL)