An Industrial Evaluation of Unit Test Generation: Finding Real Faults in a Financial Application

Automated unit test generation has been extensively
studied in the literature in recent years. Previous studies on open
source systems have shown that test generation tools are quite
effective at detecting faults, but how effective and applicable are
they in an industrial application? In this paper, we investigate this
question using a life insurance and pension products calculator
engine owned by SEB Life & Pension Holding AB Riga Branch.
To study fault-finding effectiveness, we extracted 25 real faults
from the version history of this software project, and applied
two up-to-date unit test generation tools for Java, EvoSuite and
Randoop, which implement search-based and feedback-directed
random test generation, respectively. Automatically generated test
suites detected up to 56.40% (EvoSuite) and 38.00% (Randoop) of
these faults. The analysis of our results demonstrates challenges
that need to be addressed in order to improve fault detection in
test generation tools. In particular, classification of the undetected
faults shows that 97.62% of them depend on either “specific
primitive values” (50.00%) or the construction of “complex state
configuration of objects” (47.62%).
To study applicability, we surveyed the developers of the
application under test on their experience and opinions about
the test generation tools and the generated test cases. This leads
to insights on requirements for academic prototypes for successful
technology transfer from academic research to industrial practice,
such as a need to integrate with popular build tools, and to
improve the readability of the generated tests.