![]() Matinnejad, Reza ![]() ![]() ![]() in Proceedings of 11TH JOINT MEETING OF THE EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND THE ACM SIGSOFT SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING (ESEC/FSE 2017) (2017) We present the results of applying our approach for testing Simulink controllers to one public and one proprietary model, both industrial. Our approach combines explorative and exploitative search ... [more ▼] We present the results of applying our approach for testing Simulink controllers to one public and one proprietary model, both industrial. Our approach combines explorative and exploitative search algorithms to visualize the controller behavior over its input space and to identify test scenarios in the controller input space that violate or are likely to violate the controller requirements. The engineers' feedback shows that our approach is easy to use in practice and gives them confidence about the behavior of their models. [less ▲] Detailed reference viewed: 400 (39 UL)![]() Matinnejad, Reza ![]() Doctoral thesis (2016) Context. Simulink/Stateflow is an advanced system modeling platform which is prevalently used in the Cyber Physical Systems domain, e.g., automotive industry, to implement software con- trollers. Testing ... [more ▼] Context. Simulink/Stateflow is an advanced system modeling platform which is prevalently used in the Cyber Physical Systems domain, e.g., automotive industry, to implement software con- trollers. Testing Simulink models is complex and poses several challenges to research and prac- tice. Simulink models often have mixed discrete-continuous behaviors and their correct behav- ior crucially depends on time. Inputs and outputs of Simulink models are signals, i.e., values evolving over time, rather than discrete values. Further, Simulink models are required to operate satisfactory for a large variety of hardware configurations. Finally, developing test oracles for Simulink models is challenging, particularly for requirements capturing their continuous aspects. In this dissertation, we focus on testing mixed discrete-continuous aspects of Simulink models, an important, yet not well-studied, problem. The existing Simulink testing techniques are more amenable to testing and verification of logical and state-based properties. Further, they are mostly incompatible with Simulink models containing time-continuos blocks, and floating point and non- linear computations. In addition, they often rely on the presence of formal specifications, which are expensive and rare in practice, to automate test oracles. Approach. In this dissertation, we propose a set of approaches based on meta-heuristic search and machine learning techniques to automate testing of software controllers implemented in Simulink. The work presented in this dissertation is motived by Simulink testing needs at Delphi Automotive Systems, a world leading part supplier to the automotive industry. To address the above-mentioned challenges, we rely on discrete-continuous output signals of Simulink models and provide output- based black-box test generation techniques to produce test cases with high fault-revealing ability. Our algorithms are black-box, hence, compatible with Simulink/Stateflow models in their en- tirety. Further, we do not rely on the presence of formal specifications to automate test oracles. Specifically, we propose two sets of test generation algorithms for closed-loop and open-loop con- trollers implemented in Simulink: (1) For closed-loop controllers, test oracles can be formalized and automated relying on the feedback received from the controlled system. We characterize the desired behavior of closed-loop controllers in a set of common requirements, and then use search to identify the worst-case test scenarios of the controller with respect to each requirement. (2) For open-loop controllers, we cannot automate test oracles since the feedback is not available, and test oracles are manual. Hence, we focus on providing test generation algorithms that develop small effective test suites with high fault revealing ability. We further provide a test case prioriti- zation algorithm to rank the generated test cases based on their fault revealing ability and lower the manual oracle cost. Our test generation and prioritization algorithms are evaluated with several industrial and publicly available Simulink models. Specifically, we showed that fault revealing ability of our our approach outperforms that of Simulink Design Verifier (SLDV), the only test generation toolbox of Simulink and a well-known commercial Simulink testing tool. In addition, using our approach, we were able to detect several real faults in Simulink models from our industry partner, Delphi, which had not been previously found by manual testing based on domain expertise and existing Simulink testing tools. Contributions. The main research contributions in this dissertation are: 1. An automated approach for testing closed-loop controllers that characterize the desired be- havior of such controllers in a set of common requirements, and combines random explo- ration and search to effectively identify the worst-case test scenarios of the controller with respect to each requirement. 2. An automated approach for testing highly configurable closed-loop controllers by account- ing for all their feasible configurations and providing strategies to scale the search to large multi-dimensional spaces relying on dimensionality reduction and surrogate modelling 3. A black-box output-based test generation algorithm for open-loop Simulink models which uses search to maximize the likelihood of presence of specific failure patterns (i.e., anti- patterns) in Simulink output signals. 4. A black-box output-based test generation algorithm for open-loop Simulink models that maximizes output diversity to develop small test suites with diverse output signal shapes and, hence, high fault revealing ability. 5. A test case prioritization algorithm which relies on output diversity of the generated test suites, in addition to the dynamic structural coverage achieved by individual tests, to rank test cases and help engineers identify faults faster by inspecting a few test cases. 6. Two test generation tools, namely CoCoTest and SimCoTest, that respectively implement our test generation approaches for closed-loop and open-loop controllers. [less ▲] Detailed reference viewed: 320 (36 UL)![]() Matinnejad, Reza ![]() ![]() ![]() in Proceedings of the 38th International Conference on Software Engineering (2016) Detailed reference viewed: 318 (29 UL)![]() Matinnejad, Reza ![]() ![]() ![]() in Proceedings of the 38th International Conference on Software Engineering (2016) All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches ... [more ▼] All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test gener- ation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In con- trast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink mod- els. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink. [less ▲] Detailed reference viewed: 403 (37 UL)![]() Matinnejad, Reza ![]() ![]() ![]() in Information and Software Technology (2015), 57 Context. Testing and verification of automotive embedded software is a major chal- lenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink ... [more ▼] Context. Testing and verification of automotive embedded software is a major chal- lenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink models, generating code from the models, and de- ploying the resulting code on hardware devices. Automotive software artifacts are sub- ject to three rounds of testing corresponding to the three production stages: Model-in- the-Loop (MiL), Software-in-the-Loop (SiL) and Hardware-in-the-Loop (HiL) testing. Objective. We study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed loop system. These controllers make up a large part of automotive functions, and monitor and control the operating conditions of physical devices. Method. We identify a set of requirements characterizing the behavior of continu- ous controllers, and develop a search-based technique based on random search, adap- tive random search, hill climbing and simulated annealing algorithms to automatically identify worst-case test scenarios which are utilized to generate test cases for these requirements. Results. We evaluated our approach by applying it to an industrial automotive con- troller (with 443 Simulink blocks) and to a publicly available controller (with 21 Simulink blocks). Our experience shows that automatically generated test cases lead to MiL level simulations indicating potential violations of the system requirements. Further, not only does our approach generate significantly better test cases faster than random test case generation, but it also achieves better results than test scenarios devised by domain experts. Finally, our generated test cases uncover discrepancies between envi- ronment models and the real world when they are applied at the Hardware-in-the-Loop(HiL) level. Conclusion. We propose an automated approach to MiL testing of continuous con- trollers using search. The approach is implemented in a tool and has been successfully applied to a real case study from the automotive domain. [less ▲] Detailed reference viewed: 413 (68 UL)![]() Matinnejad, Reza ![]() ![]() ![]() in Proceedings of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (2015) Modeling mixed discrete-continuous controllers using Stateflow is common practice and has a long tradition in the embedded software system industry. Testing Stateflow models is complicated by expensive ... [more ▼] Modeling mixed discrete-continuous controllers using Stateflow is common practice and has a long tradition in the embedded software system industry. Testing Stateflow models is complicated by expensive and manual test oracles that are not amenable to full automation due to the complex continuous behaviors of such models. In this paper, we reduce the cost of manual test oracles by providing test case selection algorithms that help engineers develop small test suites with high fault revealing power for Stateflow models. We present six test selection algorithms for discrete-continuous Stateflows: An adaptive random test selection algorithm that diversifies test inputs, two white-box coverage-based algorithms, a black-box algorithm that diversifies test outputs, and two search-based black-box algorithms that aim to maximize the likelihood of presence of continuous output failure patterns. We evaluate and compare our test selection algorithms, and find that our three output-based algorithms consistently outperform the coverage- and input-based algorithms in revealing faults in discrete-continuous Stateflow models. Further, we show that our output-based algorithms are complementary as the two search-based algorithms perform best in revealing specific failures with small test suites, while the output diversity algorithm is able to identify different failure types better than other algorithms when test suites are above a certain size. [less ▲] Detailed reference viewed: 355 (40 UL)![]() Matinnejad, Reza ![]() ![]() ![]() Report (2015) All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches ... [more ▼] All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test gener- ation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In con- trast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink mod- els. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink. [less ▲] Detailed reference viewed: 332 (28 UL)![]() Matinnejad, Reza ![]() ![]() ![]() in International Conference on Automated Software Engineering (ASE 2014) (2014, September) Continuous controllers have been widely used in automotive do- main to monitor and control physical components. These con- trollers are subject to three rounds of testing: Model-in-the-Loop (MiL ... [more ▼] Continuous controllers have been widely used in automotive do- main to monitor and control physical components. These con- trollers are subject to three rounds of testing: Model-in-the-Loop (MiL), Software-in-the-Loop and Hardware-in-the-Loop. In our earlier work, we used meta-heuristic search to automate MiL test- ing of fixed configurations of continuous controllers. In this paper, we extend our work to support MiL testing of all feasible configura- tions of continuous controllers. Specifically, we use a combination of dimensionality reduction and surrogate modeling techniques to scale our earlier MiL testing approach to large, multi-dimensional input spaces formed by configuration parameters. We evaluated our approach by applying it to a complex, industrial continuous controller. Our experiment shows that our approach identifies test cases indicating requirements violations. Further, we demonstrate that dimensionally reduction helps generate surrogate models with higher prediction accuracy. Finally, we show that combining our search algorithm with surrogate modelling improves its efficiency for two out of three requirements. [less ▲] Detailed reference viewed: 313 (61 UL)![]() Matinnejad, Reza ![]() ![]() ![]() in International Conference on Automated Software Engineering (ASE 2014) (2014, September) We present CoCoTest, a tool for automated testing of continuous controllers at the Model-in-the-Loop stage. CoCoTest combines explorative and exploitative search algorithms to identify scenar- ios in the ... [more ▼] We present CoCoTest, a tool for automated testing of continuous controllers at the Model-in-the-Loop stage. CoCoTest combines explorative and exploitative search algorithms to identify scenar- ios in the controller input space that violate or are likely to violate the controller requirements. This enables a scalable and systematic way to test continuous properties of such controllers. Our experi- ments show that CoCoTest identifies critical flaws in the controller design that are rarely found by manual testing and go unnoticed until late stages of embedded software system development. [less ▲] Detailed reference viewed: 271 (33 UL)![]() Matinnejad, Reza ![]() ![]() ![]() in 5th Symposium on Search-Based Software Engineering (SSBSE 2013), Springer Lecture Notes in Computer Science (2013, August) The number and the complexity of software components embedded in today’s vehicles is rapidly increasing. A large group of these components monitor and control the operating conditions of physical devices ... [more ▼] The number and the complexity of software components embedded in today’s vehicles is rapidly increasing. A large group of these components monitor and control the operating conditions of physical devices (e.g., components controlling engines, brakes, and airbags). These controllers are known as continuous controllers. In this paper, we study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed feedback loop system.We identify a set of common requirements characterizing the desired behavior of continuous controllers, and develop a search-based technique to automatically generate test cases for these requirements. We evaluated our approach by applying it to a real automotive air compressor module. Our experience shows that our approach automatically generates several test cases for which the MiL level simulations indicate potential violations of the system requirements. Further, not only do our approach generates better test cases faster than random test case generation, but we also achieve better results than test scenarios devised by domain experts. [less ▲] Detailed reference viewed: 473 (85 UL)![]() Matinnejad, Reza ![]() in International Conference on Engineering of Computer-Based Systems (ECBS 2012) (2012, April) Process-centered Software Engineering Environments, or PSEEs, are intended for the definition, modification, and enactment of software process models; they thus bring software development processes into ... [more ▼] Process-centered Software Engineering Environments, or PSEEs, are intended for the definition, modification, and enactment of software process models; they thus bring software development processes into effect. Even though research efforts in process-centered software engineering abound, PSEE technology has not received the attention that it deserves. In order to create a concise but effective and practically applicable evaluation framework for PSEEs, this paper first presents a survey of PSEEs and highlights the current state of the art of the technology. The PSEEs which have been reviewed herein have been regarded as software systems, and as such, have been characterized in terms of their requirements. After providing a conceptual critique of the scope and nature of conventional PSEEs, a detailed criteria-based evaluation of a select set of several recent PSEEs has been conducted. The evaluation criteria have been derived from PSEE requirements and the results of the critique, and have then been refined and evolved into the final criterion set. [less ▲] Detailed reference viewed: 161 (15 UL)![]() Matinnejad, Reza ![]() in International Conference on Software Engineering Research, Management and Applications (SERA2011) (2011, August) Both the model-driven and agile development approaches have significantly enhanced productivity and predictability of software development in practice. Agile Model Driven Development or AMDD is an attempt ... [more ▼] Both the model-driven and agile development approaches have significantly enhanced productivity and predictability of software development in practice. Agile Model Driven Development or AMDD is an attempt to effectively bring together the fast pace of agile development and the guaranteed quality of model-driven development. Despite the proliferation of processes claim to comply with AMDD principles, there has been little academic research examining merits and demerits of such an approach. This paper first presents the underlying theoretical foundations of AMDD and then reviews, classifies and compares existing AMDD processes. The results indicate significant disparity between compared processes that implies the AMDD concepts are still in their infancy. [less ▲] Detailed reference viewed: 165 (11 UL) |
||