Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization

With the recent advances of Deep Neural Networks (DNNs) in real-world
applications, such as Automated Driving Systems (ADS) for self-driving
cars, ensuring the reliability and safety of such DNN- enabled Systems
emerges as a fundamental topic in software testing. One of the
essential testing phases of such DNN-enabled systems is online
testing, where the system under test is embedded into a specific and
often simulated application environment (e.g., a driving environment)
and tested in a closed-loop mode in interaction with the environment.
However, despite the importance of online testing for detecting safety
violations, automatically generating new and diverse test data that
lead to safety violations present the following challenges: (1) there
can be many safety requirements to be considered at the same time, (2)
running a high-fidelity simulator is often very
computationally-intensive, and (3) the space of all possible test data
that may trigger safety violations is too large to be exhaustively
explored. In this paper, we address the challenges by proposing a
novel approach, called SAMOTA (Surrogate-Assisted Many-Objective
Testing Approach), extending existing many-objective search algorithms
for test suite generation to efficiently utilize surrogate models that
mimic the simulator, but are much less expensive to run. Empirical
evaluation results on Pylot, an advanced ADS composed of multiple
DNNs, using CARLA, a high-fidelity driving simulator, show that SAMOTA
is significantly more effective and efficient at detecting unknown
safety requirement violations than state-of-the-art many-objective
test suite generation algorithms and random search. In other words,
SAMOTA appears to be a key enabler technology for online testing in
practice.