Software; Control Engineering; Cyber-Physical Systems
Abstract :
[en] Control systems are ubiquitous and often at the core of Cyber-Physical Systems, like cars and aeroplanes. They are implemented as embedded software that interacts in closed loop with the physical world through sensors and actuators. As a consequence, the software cannot just be tested in isolation. To close the loop in a testing environment and root causing failure generated by different parts of the system, executable models are used to abstract specific components. Different testing setups can be implemented by abstracting different elements: The most common ones are model-in-the-loop, software-in-the-loop, hardware-in-the-loop, and real-physics-in-the-loop. In this article, we discuss the properties of these setups and the types of faults they can expose. We develop a comprehensive case study using the Crazyflie, a drone whose software and hardware are open source. We implement all the most common testing setups and ensure the consistent injection of faults in each of them. We inject faults in the control system and we compare with the nominal performance of the non-faulty software. Our results show the specific capabilities of the different setups in exposing faults. Contrary to intuition and previous literature, we show that the setups do not belong to a strict hierarchy, and they are best designed to maximize the differences across them rather than to be as close as possible to reality.
Research center :
Interdisciplinary Centre for Security, Reliability and Trust (SnT) > SVV - Software Verification and Validation
H2020 - 871259 - ADMORPH - Towards Adaptively Morphing Embedded Systems
Name of the research project :
ADMORPH - Towards Adaptively Morphing Embedded Systems
Funders :
European Union’s Horizon 2020 research and innovation programme Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation Union Européenne
Funding number :
871259
Funding text :
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 871259 (ADMORPH project). This publication reflects only the authors’ view and the European Com- mission is not responsible for any use that may be made of the information it contains. This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
2022. Retrieved from https://github.com/ManCla/testing-abstractions
Afsoon Afzal, Claire Le Goues, Michael Hilton, and C. Timperley. 2020. A study on challenges of testing robotic systems. In Proceedings of the IEEE 13th International Conference on Software Testing, Validation and Verification (ICST’20), 96–107.
Aldeida Aleti and Lars Grunske. 2015. Test data generation with a Kalman filter-based adaptive genetic algorithm. J. Syst. Softw. 103, C (2015), 343–352. https://doi.org/10.1016/j.jss.2014.11.035
Nadia Alshahwan, Andrea Ciancone, Mark Harman, Yue Jia, Ke Mao, Alexandru Marginean, Alexander Mols, Hila Peleg, Federica Sarro, and Ilya Zorin. 2019. Some challenges for software testing research (invited talk paper). In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’19). Association for Computing Machinery, New York, NY, USA, 1–3. https://doi.org/10.1145/3293882.3338991
Brandon Araki, John Strang, Sarah Pohorecky, Celine Qiu, Tobias Naegeli, and Daniela Rus. 2017. Multi-robot path planning for a swarm of robots that can both fly and drive. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’17). IEEE, 5575–5582. https://doi.org/10.1109/ICRA.2017.7989657
Sara Abbaspour Asadollah, Daniel Sundmark, Sigrid Eldh, and Hans Hansson. 2018. A runtime verification tool for detecting concurrency bugs in FreeRTOS embedded software. In Proceedings of the 17th International Symposium on Parallel and Distributed Computing (ISPDC’18). 172–179. https://doi.org/10.1109/ISPDC2018.2018.00032
Karl Johan Åström and Tore Hägglund. 2006. Advanced PID Control. The Instrumentation, Systems and Automation Society.
Karl Johan Åstrom and Richard M. Murray. 2008. Feedback Systems: An Introduction for Scientists and Engineers. Princeton University Press, Princeton, NJ.
Karl J. Åström and Björn Wittenmark. 2013. Computer-controlled Systems: Theory and Design. Courier Corporation.
Johannes Bach, Jacob Langner, Stefan Otten, Eric Sax, and Marc Holzäpfel. 2017. Test scenario selection for system-level verification and validation of geolocation-dependent automotive control systems. In Proceedings of the International Conference on Engineering, Technology and Innovation (ICE/ITMC). 203–210. https://doi.org/10.1109/ICE.2017. 8279890
Balaji Balasubramaniam, Hamid Bagheri, Sebastian Elbaum, and Justin Bradley. 2020. Investigating controller evolution and divergence through mining and mutation*. In Proceedings of the ACM/IEEE 11th International Conference on Cyber-Physical Systems (ICCPS’20). 151–161. https://doi.org/10.1109/ICCPS48487.2020.00022
Abhijeet Banerjee, Sudipta Chattopadhyay, and Abhik Roychoudhury. 2016. On testing embedded software. In Advances in Computers, Vol. 101. Elsevier, Amsterdam, 121–153. https://doi.org/10.1016/bs.adcom.2015.11.005
Antonia Bertolino, Pietro Braione, Guglielmo De Angelis, Luca Gazzola, Fitsum Kifetew, Leonardo Mariani, Matteo Orrù, Mauro Pezzè, Roberto Pietrantuono, Stefano Russo, and Paolo Tonella. 2021. A survey of field-based testing techniques. ACM Comput. Surv. 54, 5, Article 92 (May 2021), 39 pages. https://doi.org/10.1145/3447240
Justin M. Bradley and Hamid Bagheri. [n. d.]. Control Software: Research Directions in the Intersection of Control Theory and Software Engineering. https://doi.org/10.2514/6.2020-2102
Lionel Briand, Shiva Nejati, Mehrdad Sabetzadeh, and Domenico Bianculli. 2016. Testing the untestable: Model testing of complex software-intensive systems. In Proceedings of the 38th International Conference on Software Engineering Companion (ICSE’16). Association for Computing Machinery, New York, NY, 789–792. https://doi.org/10.1145/2889160. 2889212
Eckard Bringmann and Andreas Krämer. 2006. Systematic testing of the continuous behavior of automotive systems. In Proceedings of the International Workshop on Software Engineering for Automotive Systems (SEAS’06). Association for Computing Machinery, New York, NY, 13–20. https://doi.org/10.1145/1138474.1138479
Eckard Bringmann and Andreas Krömer. 2008. Model-based testing of automotive systems. In Proceedings of the 1st International Conference on Software Testing, Verification, and Validation. 485–493. https://doi.org/10.1109/ICST.2008.45
M. Broy, I. H. Kruger, A. Pretschner, and C. Salzmann. 2007. Engineering automotive software. Proc. IEEE 95, 2 (Feb. 2007), 356–373. https://doi.org/10.1109/JPROC.2006.888386
Barbara Barros Carlos, Tommaso Sartor, Andrea Zanelli, Gianluca Frison, Wolfram Burgard, Moritz Diehl, and Giuseppe Oriolo. 2020. An efficient real-time NMPC for quadrotor position control under communication time-delay. In Proceedings of the 16th International Conference on Control, Automation, Robotics and Vision (ICARCV’20). IEEE, 982–989. https://doi.org/10.1109/ICARCV50220.2020.9305513
Lane Desborough and Randy Miller. 2002. Increasing customer value of industrial control performance monitoring—Honeywell’ s experience. https://core.ac.uk/display/102313613
Richard Dorf and Robert Bishop. 2017. Modern Control Systems, 13th Edition.
Julian Förster. 2015. System Identification of the Crazyflie 2.0 Nano Quadrocopter.
Sergio García, Daniel Strüber, Davide Brugali, Thorsten Berger, and Patrizio Pelliccione. 2020. Robotics software engineering: A perspective from the service robotics domain. In Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’20). Association for Computing Machinery, New York, NY, 593–604. https://doi.org/10.1145/3368089.3409743
Vahid Garousi, Michael Felderer, Çağrı Murat Karapıçak, and Uğur Yılmaz. 2018. Testing embedded software: A survey of the literature. Inf. Softw. Technol. 104 (2018), 14–45. https://doi.org/10.1016/j.infsof.2018.06.016
Iris Gräßler, Eric Bodden, Jens Pottebaum, Johannes Geismann, and Daniel Roesmann. 2020. Security-oriented fault-tolerance in systems engineering: A conceptual threat modelling approach for cyber-physical production systems. In Advanced, Contemporary Control, Andrzej Bartoszewicz, Jacek Kabziński, and Janusz Kacprzyk (Eds.). Springer International Publishing, Cham, 1458–1469.
Marcus Greiff. 2017. Modelling and control of the crazyflie quadrotor for qggressive and autonomous flight by optical flow driven state estimation. https://lup.lub.lu.se/student-papers/search/publication/8905295
N. Hansen, Norbert Wiechowski, Alexander Kugler, S. Kowalewski, Thomas Rambow, and R. Busch. 2017. Model-in-the-loop and software-in-the-loop testing of closed-loop automotive software with arttest. In GI-Jahrestagung. https://dl.gi.de/items/bab4a8a8-6908-4534-92f0-2e6bbed1892f
Joachim Hänsel, Daniela Rose, Paula Herber, and Sabine Glesner. 2011. An evolutionary algorithm for the generation of timed test traces for embedded real-time systems. In Proceedings of the 4th IEEE International Conference on Software Testing, Verification and Validation. 170–179. https://doi.org/10.1109/ICST.2011.37
Janne Keränen and Tomi Räty. 2013. Validation of model-based testing in hardware in the loop platform. In Proceedings of the 10th International Conference on Information Technology: New Generations. 331–336. https://doi.org/10.1109/ITNG.2013.53
Pierre Laclau, Vladislav Tempez, Franck Ruffier, Enrico Natalizio, and Jean-Baptiste Mouret. 2021. Signal-based self-organization of a chain of UAVs for subterranean exploration. Front. Robot. AI 8 (2021), 614206. https://doi.org/10.3389/frobt.2021.614206
Frank Lamb. 2013. Industrial Automation: Hands On. McGraw–Hill. https://doi.org/10.1036/9780071816472
Klaus Lamberg, Michael Beine, Mario Eschmann, Rainer Otterbach, Mirko Conrad, and Ines Fey. 2004. Model-based testing of embedded automotive software using mtest. In SAE 2004 World Congress and Exhibition. SAE International. https://doi.org/10.4271/2004-01-1593
Edward Ashford Lee and Sanjit Arunkumar Seshia. 2016. Introduction to Embedded Systems: A Cyber-Physical Systems Approach (2nd ed.). The MIT Press.
William S. Levine. 2009. The Control Systems Handbook (2nd ed.). CRC Press.
D. Lohar, Clothilde Jeangoudoux, Joshua Sobel, Eva Darulova, and M. Christakis. 2021. A two-phase approach for conditional floating-point verification. In Tools and Algorithms for the Construction and Analysis of Systems (2021), 43–63.
Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, and Darko Marinov. 2014. An empirical analysis of flaky tests. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE’14). Association for Computing Machinery, New York, NY, 643–653. https://doi.org/10.1145/2635868.2635920
Paulo Henrique Maia, Lucas Vieira, Matheus Chagas, Yijun Yu, Andrea Zisman, and Bashar Nuseibeh. 2019. Dragonfly: A tool for simulating self-adaptive drone behaviours. In Proceedings of the IEEE/ACM 14th International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS’19). 107–113. https://doi.org/10.1109/SEAMS.2019.00022
Bogdan Marculescu, Robert Feldt, Richard Torkar, and Simon Poulding. 2015. An initial industrial evaluation of interactive search-based testing for embedded software. Appl. Soft Comput. 29 (Apr. 2015), 26–39. https://doi.org/10.1016/j.asoc.2014.12.025
Abel Marrero Perez and Stefan Kaiser. 2009. Integrating test levels for embedded systems. In Proceedings of Testing: Academic and Industrial Conference—Practice and Research Techniques (TAICPART’09). 184–193. https://doi.org/10.1109/TAICPART.2009.22
Reza Matinnejad, Shiva Nejati, Lionel Briand, and Thomas Brcukmann. 2014. MiL testing of highly configurable continuous controllers: Scalable search using surrogate models. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering (ASE’14). Association for Computing Machinery, New York, NY, 163–174. https://doi.org/10.1145/2642937.2642978
Reza Matinnejad, Shiva Nejati, and Lionel C. Briand. 2017. Automated testing of hybrid simulink/stateflow controllers: Industrial case studies. In Proceedings of the 11th Joint Meeting on Foundations of Software Engineering (ESEC/FSE’17). Association for Computing Machinery, New York, NY, 938–943. https://doi.org/10.1145/3106237.3117770
James Clerk Maxwell. 2011. On Governors. Cambridge Library Collection—Physical Sciences, Vol. 2. Cambridge University Press, 105–120. https://doi.org/10.1017/CBO9780511710377.009
Indika Meedeniya, Barbora Buhnova, Aldeida Aleti, and Lars Grunske. 2011. Reliability-driven deployment optimization for embedded systems. J. Syst. Softw. 84, 5 (2011), 835–846. https://doi.org/10.1016/j.jss.2011.01.004
Claudio Menghi, Shiva Nejati, Lionel C. Briand, and Yago Isasi Parache. 2019. Approximation-refinement testing of compute-intensive cyber-physical models: An approach based on system identification. arXiv:1910.02837. Retrieved from http://arxiv.org/abs/1910.02837
C. Menghi, P. Spoletini, M. Chechik, and C. Ghezzi. 2019. A verification-driven framework for iterative design of controllers. Formal Aspects Comput. (2019), 1–44.
Derek Mitchell, Ellen A. Cappo, and Nathan Michael. 2016. Persistent robot formation flight via online substitution. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS’16). IEEE, 4810–4815. https://doi.org/10.1109/IROS.2016.7759706
Daniel K. Molzahn, Florian Dörfler, Henrik Sandberg, Steven H. Low, Sambuddha Chakrabarti, Ross Baldick, and Javad Lavaei. 2017. A survey of distributed optimization and control algorithms for electric power systems. IEEE Trans. Smart Grid 8, 6 (2017), 2941–2962. https://doi.org/10.1109/TSG.2017.2720471
Yannick Moy, Emmanuel Ledinot, Hervé Delseny, Virginie Wiels, and Benjamin Monate. 2013. Testing or formal verification: DO-178C alternatives and industrial experience. IEEE Softw. 30, 3 (2013), 50–57. https://doi.org/10.1109/MS.2013.43
Mark W. Mueller, Michael Hamer, and Raffaello D’Andrea. 2015. Fusing ultra-wideband range measurements with accelerometers and rate gyroscopes for quadrocopter state estimation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’15). 1730–1736. https://doi.org/10.1109/ICRA.2015.7139421
Mark W. Mueller, Markus Hehn, and Raffaello D’Andrea. 2016. Covariance correction step for kalman filtering with an attitude. J. Guid. Contr. Dynam. (2016), 1–7.
Mark W. Müller and Raffaello D’Andrea. 2014. Stability and control of a quadrocopter despite the complete loss of one, two, or three propellers. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’14). IEEE, 45–52. https://doi.org/10.1109/ICRA.2014.6906588
Anitha Murugesan, Sanjai Rayadurgam, Michael W. Whalen, and Mats P. E. Heimdahl. 2015. Design considerations for modeling modes in cyber–physical systems. IEEE Des. Test 32, 5 (2015), 66–73. https://doi.org/10.1109/MDAT.2015. 2462112
Hausi A. Müller. 2017. The rise of intelligent cyber-physical systems. Computer 50, 12 (2017), 7–9. https://doi.org/10.1109/MC.2017.4451221
Shiva Nejati, Khouloud Gaaloul, Claudio Menghi, Lionel C. Briand, Stephen Foster, and David Wolfe. 2019. Evaluating model testing and model checking for finding requirements violations in simulink models. In Proceedings of the 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE’19). Association for Computing Machinery, New York, NY, 1015–1025. https://doi.org/10.1145/3338906.3340444
Luan V. Nguyen, Khaza Anuarul Hoque, Stanley Bak, Steven Drager, and Taylor T. Johnson. 2018. Cyber-physical specification mismatches. ACM Trans. Cyber-Phys. Syst. 2, 4, Article 23 (Jul’ 2018), 26 pages. https://doi.org/10.1145/ 3170500
Petter Nilsson, Omar Hussien, Ayca Balkan, Yuxiao Chen, Aaron D. Ames, Jessy W. Grizzle, Necmiye Ozay, Huei Peng, and Paulo Tabuada. 2016. Correct-by-construction adaptive cruise control: Two approaches. IEEE Trans. Contr. Syst. Technol. 24, 4 (2016), 1294–1307. https://doi.org/10.1109/TCST.2015.2501351
John-Paul Ore, Carrick Detweiler, and Sebastian Elbaum. 2018. Towards code-aware robotic simulation: Vision paper. In Proceedings of the 1st International Workshop on Robotics Software Engineering (RoSE’18). Association for Computing Machinery, New York, NY, 40–43. https://doi.org/10.1145/3196558.3196566
Jan Peleska. 2002. Hardware/Software Integration Testing for the New Airbus Aircraft Families. Retrieved from http://www.informatik.uni-bremen.de/agbs/jp/papers/peleskaTestCom2002.html https://doi.org/10.1007/978-0-387-35497-2_24
Giuseppe Silano, Emanuele Aucone, and Luigi Iannelli. 2018. CrazyS: A software-in-the-loop platform for the crazyflie 2.0 nano-quadcopter. In Proceedings of the 26th Mediterranean Conference on Control and Automation (MED’18). 1–6. https://doi.org/10.1109/MED.2018.8442759
Nathanial T. Smith, James T. Heineck, and Edward T. Schairer. 2017. Optical flow for flight and wind tunnel background oriented Schlieren imaging. https://doi.org/10.2514/6.2017-0472
Thierry Sotiropoulos, Hélène Waeselynck, Jérémie Guiochet, and Félix Ingrand. 2017. Can robot navigation bugs be found in simulation? An exploratory study. In Proceedings of the IEEE International Conference on Software Quality, Reliability and Security (QRS’17). 150–159. https://doi.org/10.1109/QRS.2017.25
Gerald Steinbauer. 2013. A survey about faults of robots used in RoboCup. In RoboCup 2012: Robot Soccer World Cup XVI, Xiaoping Chen, Peter Stone, Luis Enrique Sucar, and Tijn van der Zant (Eds.). Springer, Berlin, 344–355.
Andrea Stocco, Brian Pulfer, and Paolo Tonella. 2023. Mind the gap! A study on the transferability of virtual versus physical-world testing of autonomous driving systems. IEEE Trans. Softw. Eng. 49, 4 (2023), 1928–1940. https://doi.org/10.1109/TSE.2022.3202311
James R. Taylor, Evan M. Drumwright, and Gabriel Parmer. 2014. Making time make sense in robotic simulation. In Proceedings of the 4th International Conference on Simulation, Modeling, and Programming for Autonomous Robots, Volume 8810 (SIMPAR’14). Springer-Verlag, Berlin, 1–12. https://doi.org/10.1007/978-3-319-11900-7_1
James R. Taylor, Evan M. Drumwright, and Gabriel Parmer. 2014. Temporally consistent simulation of robots and their controllers. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference. https://doi.org/10.1115/DETC2014-35609
C. S. Timperley, A. Afzal, D. S. Katz, J. M. Hernandez, and C. Le Goues. 2018. Crashing simulated planes is cheap: Can simulation detect robotics bugs early? In Proceedings of the IEEE 11th International Conference on Software Testing, Verification and Validation (ICST’18). 331–342. https://doi.org/10.1109/ICST.2018.00040
R. M. van der Knijff. 2014. Control systems/SCADA forensics, what’s the difference? Digit. Invest. 11, 3 (2014), 160–174. https://doi.org/10.1016/j.diin.2014.06.007
Gang Wang, Weixin Yang, Na Zhao, Yunfeng Ji, Yantao Shen, Hao Xu, and Peng Li. 2020. Distributed consensus control of multiple UAVs in a constrained environment. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA’20). IEEE, 3234–3240. https://doi.org/10.1109/ICRA40945.2020.9196926
Michael W. Whalen, Anitha Murugesan, Sanjai Rayadurgam, and Mats P. E. Heimdahl. 2014. Structuring simulink models for verification and reuse. In Proceedings of the 6th International Workshop on Modeling in Software Engineering (MiSE’14). Association for Computing Machinery, New York, NY, 19–24. https://doi.org/10.1145/2593770.2593776
A. L. White. 2001. Comments on modified condition/decision coverage for software testing [of flight control software]. In Proceedings of the IEEE Aerospace Conference Proceedings, Vol. 6. 2821–2827. https://doi.org/10.1109/AERO.2001. 931302
Johannes Wienke, Sebastian Meyer zu Borgsen, and Sebastian Wrede. 2016. A data set for fault detection research on component-based robotic systems. In Towards Autonomous Robotic Systems, Lyuba Alboul, Dana Damian, and Jonathan M. Aitken (Eds.). Springer International Publishing, Cham, 339–350.
Justyna Zander, Ina Schieferdecker, and Pieter Mosterman. 2011. Model-Based Testing for Embedded Systems.
Xi Zheng, Christine Julien, Miryung Kim, and Sarfraz Khurshid. 2017. Perceptions on the state of the art in verification and validation in cyber-physical systems. IEEE Syst. J. 11, 4 (2017), 2614–2627. https://doi.org/10.1109/JSYST.2015. 2496293
Michael Zimmer, J. Hedrick, and Edward A. Lee. 2015. Ramifications of software implementation and deployment: A case study on yaw moment controller design. In Proceedings of the American Control Conference (ACC’15). 2014–2019.
