Reference : Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Ob...
Scientific congresses, symposiums and conference proceedings : Paper published in a book
Engineering, computing & technology : Computer science
Security, Reliability and Trust
http://hdl.handle.net/10993/50091
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and Many-Objective Optimization
English
Ul Haq, Fitash mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
Shin, Donghwan mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
Briand, Lionel mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
May-2022
Proceedings of the 44th International Conference on Software Engineering (ICSE ’22)
ACM
811-822
Yes
No
International
New York, NY
USA
44th International Conference on Software Engineering (ICSE ’22)
from 21-05-2022 to 29-05-2022
[en] DNN testing ; self-driving cars ; online testing ; many-objective search ; surrogate-assisted optimization
[en] With the recent advances of Deep Neural Networks (DNNs) in real-world
applications, such as Automated Driving Systems (ADS) for self-driving
cars, ensuring the reliability and safety of such DNN- enabled Systems
emerges as a fundamental topic in software testing. One of the
essential testing phases of such DNN-enabled systems is online
testing, where the system under test is embedded into a specific and
often simulated application environment (e.g., a driving environment)
and tested in a closed-loop mode in interaction with the environment.
However, despite the importance of online testing for detecting safety
violations, automatically generating new and diverse test data that
lead to safety violations present the following challenges: (1) there
can be many safety requirements to be considered at the same time, (2)
running a high-fidelity simulator is often very
computationally-intensive, and (3) the space of all possible test data
that may trigger safety violations is too large to be exhaustively
explored. In this paper, we address the challenges by proposing a
novel approach, called SAMOTA (Surrogate-Assisted Many-Objective
Testing Approach), extending existing many-objective search algorithms
for test suite generation to efficiently utilize surrogate models that
mimic the simulator, but are much less expensive to run. Empirical
evaluation results on Pylot, an advanced ADS composed of multiple
DNNs, using CARLA, a high-fidelity driving simulator, show that SAMOTA
is significantly more effective and efficient at detecting unknown
safety requirement violations than state-of-the-art many-objective
test suite generation algorithms and random search. In other words,
SAMOTA appears to be a key enabler technology for online testing in
practice.
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > Software Verification and Validation Lab (SVV Lab)
Fonds National de la Recherche - FnR
http://hdl.handle.net/10993/50091
10.1145/3510003.3510188
H2020 ; 694277 - TUNE - Testing the Untestable: Model Testing of Complex Software-Intensive Systems
FnR ; FNR14711346 > Fabrizio Pastore > FUNTASY > Functional Safety For Autonomous Systems > 01/08/2020 > 31/07/2023 > 2020

File(s) associated to this reference

Fulltext file(s):

FileCommentaryVersionSizeAccess
Open access
SAMOTA-authorversion.pdfAuthor postprint821.43 kBView/Open

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.