Dissertations and theses : Doctoral thesis
Engineering, computing & technology : Computer science
Security, Reliability and Trust
Ul Haq, Fitash mailto [University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV >]
University of Luxembourg, ​​Luxembourg
Docteur en Informatique
Briand, Lionel mailto
Aouada, Djamila mailto
Pastore, Fabrizio mailto
Tonella, Paolo mailto
Gambi, Alessio mailto
[en] Search-based Software Engineering ; Software Testing ; DNN Testing ; DADS testing ; Online Testing ; Offline Testing ; Surrogate Assisted Optimisation ; Many-Objective Optimisation
[en] With the recent advances of Deep Neural Networks (DNNs) in real-world applications, such as Automated Driving Systems (ADS) for self-driving cars, ensuring the reliability and safety of such DNN-Enabled Systems (DES) emerges as a fundamental topic in software testing. Automatically generating new and diverse test data that lead to safety violations of DES presents the following challenges: (1) there can be many safety requirements to be considered at the same time, (2) running a high-fidelity simulator is often very computationally intensive, (3) the space of all possible test data that may trigger safety violations is too large to be exhaustively explored, (4) depending upon the accuracy of the DES under test, it may be infeasible to find a scenario causing violations for some requirements, and (5) DNNs are often developed by a third party, who does not provide access to internal information of the DNNs.
In this dissertation, in collaboration with IEE sensing, we address the aforementioned challenges by providing scalable and practical automated solutions for testing Deep Learning (DL) models and systems. Specifically, we present the following in the dissertation.
1. We conduct an empirical study to compare offline testing and online testing in the context
of Automated Driving Systems (ADS). We also investigate whether simulator-generated data can be used in lieu of real-world data. Furthermore, we investigate whether offline testing results can be used to help reduce the cost of online testing.
2. We propose an approach to generate test data using many-objective search algorithms tailored for test suite generation to generate test data for DNN with many outputs. We also demonstrate a way to learn conditions that cause the DNN to mispredict the outputs.
3. In order to reduce the number of computationally expensive simulations, we propose an automated approach, SAMOTA, to generate data for DNN-enabled automated driving systems, using many- objective search and surrogate-assisted optimisation.
4. The environmental conditions (e.g., weather, lighting) often stay the same during a simulation, which can limit the scope of testing. To address this limitation, we present an automated approach, MORLAT, to dynamically interact with the environment during simulation. MORLAT relies on reinforcement learning and many-objective optimisation.
We evaluate our approaches using state-of-the-art deep neural networks and systems. The results show that our approaches perform statistically better than the alternatives
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
Researchers ; Professionals ; Students
H2020 ; 694277 - TUNE - Testing the Untestable: Model Testing of Complex Software-Intensive Systems
FnR ; FNR14711346 > Fabrizio Pastore > FUNTASY > Functional Safety For Autonomous Systems > 01/08/2020 > 31/07/2023 > 2020

File(s) associated to this reference

Fulltext file(s):

Open access
Scalable_and_Practical_Automated_Testing_of_Deep_Learning_Models_and_Systems-final-version.pdfAuthor postprint3.79 MBView/Open

Bookmark and Share SFX Query

All documents in ORBilu are protected by a user license.