Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems

[en] Deep Neural Networks (DNNs) have been widely used to perform real-world tasks in cyber-physical systems such as Autonomous Driving Systems (ADS). Ensuring the correct behavior of such DNN-Enabled Systems (DES) is a crucial topic. Online testing is one of the promising modes for testing such systems with their application environments (simulated or real) in a closed loop, taking into account the continuous interaction between the systems and their environments. However, the environmental variables (e.g., lighting conditions) that might change during the systems' operation in the real world, causing the DES to violate requirements (safety, functional), are often kept constant during the execution of an online test scenario due to the two major challenges: (1) the space of all possible scenarios to explore would become even larger if they changed and (2) there are typically many requirements to test simultaneously. In this paper, we present MORLOT (Many-Objective Reinforcement Learning for Online Testing), a novel online testing approach to address these challenges by combining Reinforcement Learning (RL) and many-objective search. MORLOT leverages RL to incrementally generate sequences of environmental changes while relying on many-objective search to determine the changes so that they are more likely to achieve any of the uncovered objectives. We empirically evaluate MORLOT using CARLA, a high-fidelity simulator widely used for autonomous driving research, integrated with Transfuser, a DNN-enabled ADS for end-to-end driving. The evaluation results show that MORLOT is significantly more effective and efficient than alternatives with a large effect size. In other words, MORLOT is a good option to test DES with dynamically changing environments while accounting for multiple safety requirements.

Research center :

Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation

Disciplines :

Computer science

Author, co-author :

UL HAQ, Fitash ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV

SHIN, Donghwan ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV

BRIAND, Lionel ; University of Luxembourg > Interdisciplinary Centre for Security, Reliability and Trust (SNT) > SVV

External co-authors :

yes

Language :

English

Title :

Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems

Publication date :

May 2023

Event name :

45th International Conference on Software Engineering (ICSE ’23)

Event date :

from 14-05-2023 to 20-05-2023

Audience :

International

Main work title :

45th International Conference on Software Engineering (ICSE ’23)

Publisher :

ACM, New York, NY, United States

Peer reviewed :

Peer reviewed

Focus Area :

Security, Reliability and Trust

Available on ORBilu :

since 21 August 2023

Statistics

Number of views

152 (8 by Unilu)

Number of downloads

80 (2 by Unilu)

More statistics

Scopus citations^®

Scopus citations^®
without self-citations

OpenAlex citations

Bibliography

A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks, " Advances in neural information processing systems, vol. 25, 2012.
G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Contextdependent pre-trained deep neural networks for largevocabulary speech recognition, " IEEE Transactions on audio, speech, and language processing, vol. 20, no. 1, pp. 30-42, 2011.
C. Szegedy, A. Toshev, and D. Erhan, "Deep neural networks for object detection, " Advances in neural information processing systems, vol. 26, 2013.
J. M. Zhang, M. Harman, L. Ma, and Y. Liu, "Machine learning testing: Survey, landscapes and horizons, " IEEE Transactions on Software Engineering, vol. 48, no. 1, pp. 1-36, 2022.
F. U. Haq, D. Shin, S. Nejati, and L. C. Briand, "Can offline testing of deep neural networks replace their online testing?" CoRR, vol. abs/2101. 11118, 2021. [Online]. Available: https://arxiv. org/abs/2101. 11118
A. Gambi, M. Mueller, and G. Fraser, "Automatically testing self-driving cars with search-based procedural content generation, " in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 318-328. [Online]. Available: https://doi. org/10. 1145/3293882. 3330566
F. Ul Haq, D. Shin, and L. Briand, "Efficient online testing for dnn-enabled systems using surrogate-assisted and many-objective optimization, " in Proceedings of the 44th International Conference on Software Engineering (ICSE'22). ACM, 2022.
V. Riccio and P. Tonella, "Model-based exploration of the frontier of behaviours for deep learning system testing, " in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 876-888. [Online]. Available: https://doi. org/10. 1145/3368089. 3409730
M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer, "Adaptive stress testing for autonomous vehicles, " in 2018 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2018, pp. 1-7.
C. Lu, Y. Shi, H. Zhang, M. Zhang, T. Wang, T. Yue, and S. Ali, "Learning configurations of operating environment of autonomous vehicles to maximize their collisions, " IEEE Transactions on Software Engineering, 2022.
R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018.
A. Panichella, F. M. Kifetew, and P. Tonella, "Reformulating branch coverage as a many-objective optimization problem, " in 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), 2015, pp. 1-10.
R. B. Abdessalem, A. Panichella, S. Nejati, L. C. Briand, and T. Stifter, "Testing autonomous cars for feature interaction failures using many-objective search, " in Proceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering, ser. ASE 2018. New York, NY, USA: Association for Computing Machinery, 2018, p. 143-154. [Online]. Available: https://doi. org/10. 1145/3238147. 3238192
A. Prakash, K. Chitta, and A. Geiger, "Multi-modal fusion transformer for end-to-end autonomous driving, " in Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
C. Team, "CARLA Autonomous Driving Leaderboard, " 2022. [Online]. Available: https://leaderboard. carla. org/leaderboard/
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, "CARLA: An open urban driving simulator, " in Proceedings of the 1st Annual Conference on Robot Learning, ser. Proceedings of Machine Learning Research, S. Levine, V. Vanhoucke, and K. Goldberg, Eds., vol. 78. PMLR, 13-15 Nov 2017, pp. 1-16. [Online]. Available: https://proceedings. mlr. press/v78/dosovitskiy17a. html
C. J. Watkins and P. Dayan, "Q-learning, " Machine learning, vol. 8, no. 3, pp. 279-292, 1992.
G. A. Rummery and M. Niranjan, On-line Q-learning using connectionist systems. Citeseer, 1994, vol. 37.
K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath, "Deep reinforcement learning: A brief survey, " IEEE Signal Processing Magazine, vol. 34, no. 6, pp. 26-38, 2017.
H. Ishibuchi, N. Tsukamoto, and Y. Nojima, "Evolutionary many-objective optimization: A short review, " in 2008 IEEE Congress on Evolutionary Computation (IEEE World Congress on Computational Intelligence), 2008, pp. 2419-2426.
S. Chand and M. Wagner, "Evolutionary many-objective optimization: A quick-start guide, " Surveys in Operations Research and Management Science, vol. 20, no. 2, pp. 35-42, 2015.
Q. Zhang and H. Li, "Moea/d: A multiobjective evolutionary algorithm based on decomposition, " IEEE Transactions on Evolutionary Computation, vol. 11, no. 6, pp. 712-731, 2007.
K. Deb and H. Jain, "An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints, " IEEE transactions on evolutionary computation, vol. 18, no. 4, pp. 577-601, 2013.
M. Klischat and M. Althoff, "Generating critical test scenarios for automated vehicles with evolutionary algorithms, " in 2019 IEEE Intelligent Vehicles Symposium (IV), 2019, pp. 2352-2358.
A. Calò, P. Arcaini, S. Ali, F. Hauer, and F. Ishikawa, "Generating avoidable collision scenarios for testing autonomous driving systems, " in 2020 IEEE 13th International Conference on Software Testing, Validation and Verification (ICST), 2020, pp. 375-386.
R. B. Abdessalem, S. Nejati, L. C. Briand, and T. Stifter, "Testing vision-based control systems using learnable evolutionary algorithms, " in 2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE). IEEE, 2018, pp. 1016-1026.
G. Li, Y. Li, S. Jha, T. Tsai, M. Sullivan, S. K. S. Hari, Z. Kalbarczyk, and R. Iyer, "Av-fuzzer: Finding safety violations in autonomous driving systems, " in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2020, pp. 25-36.
R. Lee, M. J. Kochenderfer, O. J. Mengshoel, G. P. Brat, and M. P. Owen, "Adaptive stress testing of airborne collision avoidance systems, " in 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC). IEEE, 2015, pp. 6C2-1.
A. Corso, P. Du, K. Driggs-Campbell, and M. J. Kochenderfer, "Adaptive stress testing with reward augmentation for autonomous vehicle validatio, " in 2019 IEEE Intelligent Transportation Systems Conference (ITSC). IEEE, 2019, pp. 163-168.
A. Sharif and D. Marijan, "Adversarial deep reinforcement learning for trustworthy autonomous driving policies, " arXiv preprint arXiv: 2112. 11937, 2021.
D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, "A general reinforcement learning algorithm that masters chess, shogi, and go through self-play, " Science, vol. 362, no. 6419, pp. 1140-1144, 2018. [Online]. Available: https://www. science. org/doi/abs/10. 1126/science. aar6404
Y. Zheng, Y. Liu, X. Xie, Y. Liu, L. Ma, J. Hao, and Y. Liu, "Automatic web testing using curiosity-driven reinforcement learning, " in 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 2021, pp. 423-435.
Y. Zheng, X. Xie, T. Su, L. Ma, J. Hao, Z. Meng, Y. Liu, R. Shen, Y. Chen, and C. Fan, "Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning, " in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 772-784.
M. Pan, A. Huang, G. Wang, T. Zhang, and X. Li, "Reinforcement learning based curiosity-driven testing of android applications, " in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, 2020, pp. 153-164.
K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need, " Advances in neural information processing systems, vol. 30, 2017.
E. Leurent, "A survey of state-action representations for autonomous driving, " 2018.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al., "Human-level control through deep reinforcement learning, " nature, vol. 518, no. 7540, pp. 529-533, 2015.
R. Yang, X. Sun, and K. Narasimhan, "A generalized algorithm for multi-objective reinforcement learning and policy adaptation, " Advances in Neural Information Processing Systems, vol. 32, 2019.
H. B. Mann and D. R. Whitney, "On a test of whether one of two random variables is stochastically larger than the other, " The annals of mathematical statistics, pp. 50-60, 1947.
A. Vargha and H. D. Delaney, "A critique and improvement of the cl common language effect size statistics of mcgraw and wong, " Journal of Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101-132, 2000. [Online]. Available: https://doi. org/10. 3102/10769986025002101
F. U. Haq, D. Shin, and L. Briand, "Replication Package for Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems, " 12 2022. [Online]. Available: https://doi. org/10. 6084/m9. figshare. 20526867