References of "Nejati, Shiva 50002745"
     in
Bookmark and Share    
Full Text
Peer Reviewed
See detailSimulink Fault Localisation: an Iterative Statistical Debugging Approach
Liu, Bing UL; Lucia, Lucia UL; Nejati, Shiva UL et al

in Software Testing, Verification & Reliability (2016), 26(6), 431-459

Debugging Simulink models presents a significant challenge in the embedded industry. In this work, we propose SimFL, a fault localization approach for Simulink models by combining statistical debugging ... [more ▼]

Debugging Simulink models presents a significant challenge in the embedded industry. In this work, we propose SimFL, a fault localization approach for Simulink models by combining statistical debugging and dynamic model slicing. Simulink models, being visual and hierarchical, have multiple outputs at different hierarchy levels. Given a set of outputs to observe for localizing faults, we generate test execution slices, for each test case and output, of the Simulink model. In order to further improve fault localization accuracy, we propose iSimFL, an iterative fault localization algorithm. At each iteration, iSimFL increases the set of observable outputs by including outputs at lower hierarchy levels, thus increasing the test oracle cost but offsetting it with significantly more precise fault localization. We utilize a heuristic stopping criterion to avoid unnecessary test oracle extension. We evaluate our work on three industrial Simulink models from Delphi Automotive. Our results show that, on average, SimFL ranks faulty blocks in the top 8.9% in the list of suspicious blocks. Further, we show that iSimFL significantly improves this percentage down to 4.4% by requiring engineers to observe only an average of five additional outputs at lower hierarchy levels on top of high-level model outputs. [less ▲]

Detailed reference viewed: 290 (57 UL)
Full Text
Peer Reviewed
See detailTesting the Untestable: Model Testing of Complex Software-Intensive Systems
Briand, Lionel UL; Nejati, Shiva UL; Sabetzadeh, Mehrdad UL et al

in Proceedings of the 38th International Conference on Software Engineering (ICSE 2016) Companion (2016, May)

Detailed reference viewed: 618 (50 UL)
Full Text
Peer Reviewed
See detailLocalizing Multiple Faults in Simulink Models.
Liu, Bing UL; Lucia, Lucia UL; Nejati, Shiva UL et al

in 23rd IEEE International Conference on Software Analysis, Evolution, and Reengineering (SANER 2016) (2016)

As Simulink is a widely used language in the embedded industry, there is a growing need to support debugging activities for Simulink models. In this work, we propose an approach to localize multiple ... [more ▼]

As Simulink is a widely used language in the embedded industry, there is a growing need to support debugging activities for Simulink models. In this work, we propose an approach to localize multiple faults in Simulink models. Our approach builds on statistical debugging and is iterative. At each iteration, we identify and resolve one fault and re-test models to focus on localizing faults that might have been masked before. We use decision trees to cluster together failures that satisfy similar (logical) conditions on model blocks or inputs. We then present two alternative selection criteria to choose a cluster that is more likely to yield the best fault localization results among the clusters produced by our decision trees. Engineers are expected to inspect the ranked list obtained from the selected cluster to identify faults. We evaluate our approach on 240 multi-fault models obtained from three different industrial subjects. We compare our approach with two baselines: (1) Statistical debugging without clustering, and (2) State-of-the-art clustering-based statistical debugging. Our results show that our approach significantly reduces the number of blocks that engineers need to inspect in order to localize all faults, when compared with the two baselines. Furthermore, with our approach, there is less performance degradation than in the baselines when increasing the number of faults in the underlying models. [less ▲]

Detailed reference viewed: 341 (41 UL)
Full Text
Peer Reviewed
See detailTesting Advanced Driver Assistance Systems using Multi-objective Search and Neural Networks
Ben Abdessalem (helali), Raja UL; Nejati, Shiva UL; Briand, Lionel UL et al

in International Conference on Automated Software Engineering (ASE 2016) (2016)

Recent years have seen a proliferation of complex Advanced Driver Assistance Systems (ADAS), in particular, for use in autonomous cars. These systems consist of sensors and cameras as well as image ... [more ▼]

Recent years have seen a proliferation of complex Advanced Driver Assistance Systems (ADAS), in particular, for use in autonomous cars. These systems consist of sensors and cameras as well as image processing and decision support software components. They are meant to help drivers by providing proper warnings or by preventing dangerous situations. In this paper, we focus on the problem of design time testing of ADAS in a simulated environment. We provide a testing approach for ADAS by combining multi- objective search with surrogate models developed based on neural networks. We use multi-objective search to guide testing towards the most critical behaviors of ADAS. Surrogate modeling enables our testing approach to explore a larger part of the input search space within limited computational resources. We characterize the condition under which the multi-objective search algorithm behaves the same with and without surrogate modeling, thus showing the accuracy of our approach. We evaluate our approach by applying it to an industrial ADAS system. Our experiment shows that our approach automatically identifies test cases indicating critical ADAS behaviors. Further, we show that combining our search algorithm with surrogate modeling improves the quality of the generated test cases, especially under tight and realistic computational resources. [less ▲]

Detailed reference viewed: 599 (135 UL)
Full Text
Peer Reviewed
See detailAutomated Test Suite Generation for Time-Continuous Simulink Models
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL et al

in Proceedings of the 38th International Conference on Software Engineering (2016)

All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches ... [more ▼]

All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test gener- ation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In con- trast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink mod- els. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink. [less ▲]

Detailed reference viewed: 342 (34 UL)
Full Text
Peer Reviewed
See detailAutomated Change Impact Analysis between SysML Models of Requirements and Design
Nejati, Shiva UL; Sabetzadeh, Mehrdad UL; Arora, Chetan UL et al

in 24th ACM SIGSOFT International Symposium on the Foundations of Software Engineering, Seattle 13-18 November 2016 (2016)

An important activity in systems engineering is analyzing how a change in requirements will impact the design of a system. Performing this analysis manually is expensive, particularly for complex systems ... [more ▼]

An important activity in systems engineering is analyzing how a change in requirements will impact the design of a system. Performing this analysis manually is expensive, particularly for complex systems. In this paper, we propose an approach to automatically identify the impact of requirements changes on system design, when the requirements and design elements are expressed using models. We ground our approach on the Systems Modeling Language (SysML) due to SysML’s increasing use in industrial applications. Our approach has two steps: For a given change, we first apply a static slicing algorithm to extract an estimated set of impacted model elements. Next, we rank the elements of the resulting set according to a quantitative measure designed to predict how likely it is for each element to be impacted. The measure is computed using Natural Language Processing (NLP) applied to the textual content of the elements. Engineers can then inspect the ranked list of elements and identify those that are actually impacted. We evaluate our approach on an industrial case study with 16 real-world requirements changes. Our results suggest that, using our approach, engineers need to inspect on average only 4.8% of the entire design in order to identify the actually-impacted elements. We further show that our results consistently improve when our analysis takes into account both structural and behavioral diagrams rather than only structural ones, and the natural-language content of the diagrams in addition to only their structural and behavioral content. [less ▲]

Detailed reference viewed: 320 (30 UL)
Full Text
Peer Reviewed
See detailSimCoTest: A Test Suite Generation Tool for Simulink/Stateflow Controllers
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL et al

in Proceedings of the 38th International Conference on Software Engineering (2016)

Detailed reference viewed: 243 (26 UL)
Full Text
Peer Reviewed
See detailCombining Genetic Algorithms and Constraint Programming to Support Stress Testing of Task Deadlines
DI ALESIO, Stefano; Briand, Lionel UL; Nejati, Shiva UL et al

in ACM Transactions on Software Engineering & Methodology (2015), 25(1),

Tasks in Real Time Embedded Systems (RTES) are often subject to hard deadlines, that constrain how quickly the system must react to external inputs. These inputs and their timing vary in a large domain ... [more ▼]

Tasks in Real Time Embedded Systems (RTES) are often subject to hard deadlines, that constrain how quickly the system must react to external inputs. These inputs and their timing vary in a large domain depending on the environment state, and can never be fully predicted prior to system execution. Therefore, approaches for stress testing must be developed to uncover possible deadline misses of tasks for different input arrival times. In this paper, we describe stress test case generation as a search problem over the space of task arrival times. Specifically, we search for worst case scenarios maximizing deadline misses where each scenario characterizes a test case. In order to scale our search to large industrial-size problems, we combine two state-of-the-art search strategies, namely Genetic Algorithms (GA) and Constraint Programming (CP). Our experimental results show that, in comparison with GA and CP in isolation, GA+CP achieves nearly the same effectiveness as CP and the same efficiency and solution diversity as GA, thus combining the advantages of the two strategies. In light of these results, we conclude that a combined GA+CP approach to stress testing is more likely to scale to large and complex systems. [less ▲]

Detailed reference viewed: 487 (53 UL)
Full Text
Peer Reviewed
See detailSearch-Based Automated Testing of Continuous Controllers: Framework, Tool Support, and Case Studies
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL et al

in Information and Software Technology (2015), 57

Context. Testing and verification of automotive embedded software is a major chal- lenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink ... [more ▼]

Context. Testing and verification of automotive embedded software is a major chal- lenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink models, generating code from the models, and de- ploying the resulting code on hardware devices. Automotive software artifacts are sub- ject to three rounds of testing corresponding to the three production stages: Model-in- the-Loop (MiL), Software-in-the-Loop (SiL) and Hardware-in-the-Loop (HiL) testing. Objective. We study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed loop system. These controllers make up a large part of automotive functions, and monitor and control the operating conditions of physical devices. Method. We identify a set of requirements characterizing the behavior of continu- ous controllers, and develop a search-based technique based on random search, adap- tive random search, hill climbing and simulated annealing algorithms to automatically identify worst-case test scenarios which are utilized to generate test cases for these requirements. Results. We evaluated our approach by applying it to an industrial automotive con- troller (with 443 Simulink blocks) and to a publicly available controller (with 21 Simulink blocks). Our experience shows that automatically generated test cases lead to MiL level simulations indicating potential violations of the system requirements. Further, not only does our approach generate significantly better test cases faster than random test case generation, but it also achieves better results than test scenarios devised by domain experts. Finally, our generated test cases uncover discrepancies between envi- ronment models and the real world when they are applied at the Hardware-in-the-Loop(HiL) level. Conclusion. We propose an automated approach to MiL testing of continuous con- trollers using search. The approach is implemented in a tool and has been successfully applied to a real case study from the automotive domain. [less ▲]

Detailed reference viewed: 350 (65 UL)
Full Text
See detailAutomated Test Suite Generation for Time-Continuous Simulink Models
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL

Report (2015)

All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches ... [more ▼]

All engineering disciplines are founded and rely on models, al- though they may differ on purposes and usages of modeling. Inter- disciplinary domains such as Cyber Physical Systems (CPSs) seek approaches that incorporate different modeling needs and usages. Specifically, the Simulink modeling platform greatly appeals to CPS engineers due to its seamless support for simulation and code generation. In this paper, we propose a test generation approach that is applicable to Simulink models built for both purposes of simulation and code generation. We define test inputs and outputs as signals that capture evolution of values over time. Our test gener- ation approach is implemented as a meta-heuristic search algorithm and is guided to produce test outputs with diverse shapes according to our proposed notion of diversity. Our evaluation, performed on industrial and public domain models, demonstrates that: (1) In con- trast to the existing tools for testing Simulink models that are only applicable to a subset of code generation models, our approach is applicable to both code generation and simulation Simulink mod- els. (2) Our new notion of diversity for output signals outperforms random baseline testing and an existing notion of signal diversity in revealing faults in Simulink models. (3) The fault revealing ability of our test generation approach outperforms that of the Simulink Design Verifier, the only testing toolbox for Simulink. [less ▲]

Detailed reference viewed: 261 (28 UL)
Full Text
Peer Reviewed
See detailEffective Test Suites for Mixed Discrete-Continuous Stateflow Controllers
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL et al

in Proceedings of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (2015)

Modeling mixed discrete-continuous controllers using Stateflow is common practice and has a long tradition in the embedded software system industry. Testing Stateflow models is complicated by expensive ... [more ▼]

Modeling mixed discrete-continuous controllers using Stateflow is common practice and has a long tradition in the embedded software system industry. Testing Stateflow models is complicated by expensive and manual test oracles that are not amenable to full automation due to the complex continuous behaviors of such models. In this paper, we reduce the cost of manual test oracles by providing test case selection algorithms that help engineers develop small test suites with high fault revealing power for Stateflow models. We present six test selection algorithms for discrete-continuous Stateflows: An adaptive random test selection algorithm that diversifies test inputs, two white-box coverage-based algorithms, a black-box algorithm that diversifies test outputs, and two search-based black-box algorithms that aim to maximize the likelihood of presence of continuous output failure patterns. We evaluate and compare our test selection algorithms, and find that our three output-based algorithms consistently outperform the coverage- and input-based algorithms in revealing faults in discrete-continuous Stateflow models. Further, we show that our output-based algorithms are complementary as the two search-based algorithms perform best in revealing specific failures with small test suites, while the output diversity algorithm is able to identify different failure types better than other algorithms when test suites are above a certain size. [less ▲]

Detailed reference viewed: 299 (39 UL)
Full Text
See detailSimulink Fault Localization: an Iterative Statistical Debugging Approach
Liu, Bing UL; Lucia, Lucia UL; Nejati, Shiva UL et al

Report (2015)

Debugging Simulink models presents a significant challenge in the embedded industry. In this work, we propose SimFL, a fault localization approach for Simulink models by combining statistical debugging ... [more ▼]

Debugging Simulink models presents a significant challenge in the embedded industry. In this work, we propose SimFL, a fault localization approach for Simulink models by combining statistical debugging and dynamic model slicing. Simulink models, being visual and hierarchical, have multiple outputs at different hierarchy levels. Given a set of outputs to observe for localizing faults, we generate test execution slices, for each test case and output, of the Simulink model. In order to further improve fault localization accuracy, we propose iSimFL, an iterative fault localization algorithm. At each iteration, iSimFL increases the set of observable outputs by including outputs at lower hierarchy levels, thus increasing the test oracle cost but offsetting it with significantly more precise fault localization. We utilize a heuristic stopping criterion to avoid unnecessary test oracle extension. We evaluate our work on three industrial Simulink models from Delphi Automotive. Our results show that, on average, SimFL ranks faulty blocks in the top 8.9% in the list of suspicious blocks. Further, we show that iSimFL significantly improves this percentage down to 4.4% by requiring engineers to observe only an average of five additional outputs at lower hierarchy levels on top of high-level model outputs. [less ▲]

Detailed reference viewed: 318 (29 UL)
Full Text
Peer Reviewed
See detailMiL Testing of Highly Configurable Continuous Controllers: Scalable Search Using Surrogate Models
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL et al

in International Conference on Automated Software Engineering (ASE 2014) (2014, September)

Continuous controllers have been widely used in automotive do- main to monitor and control physical components. These con- trollers are subject to three rounds of testing: Model-in-the-Loop (MiL ... [more ▼]

Continuous controllers have been widely used in automotive do- main to monitor and control physical components. These con- trollers are subject to three rounds of testing: Model-in-the-Loop (MiL), Software-in-the-Loop and Hardware-in-the-Loop. In our earlier work, we used meta-heuristic search to automate MiL test- ing of fixed configurations of continuous controllers. In this paper, we extend our work to support MiL testing of all feasible configura- tions of continuous controllers. Specifically, we use a combination of dimensionality reduction and surrogate modeling techniques to scale our earlier MiL testing approach to large, multi-dimensional input spaces formed by configuration parameters. We evaluated our approach by applying it to a complex, industrial continuous controller. Our experiment shows that our approach identifies test cases indicating requirements violations. Further, we demonstrate that dimensionally reduction helps generate surrogate models with higher prediction accuracy. Finally, we show that combining our search algorithm with surrogate modelling improves its efficiency for two out of three requirements. [less ▲]

Detailed reference viewed: 255 (58 UL)
Full Text
Peer Reviewed
See detailWorst-case Scheduling of Software Tasks: A Constraint Optimization Model to Support Performance Testing
Di Alesio, Stefano UL; Nejati, Shiva UL; Briand, Lionel UL et al

in International Conference on Principles and Practice of Constraint Programming (2014, September)

Detailed reference viewed: 146 (12 UL)
Full Text
Peer Reviewed
See detailCoCoTest: A Tool for Model-in-the-Loop Testing of Continuous Controller
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL et al

in International Conference on Automated Software Engineering (ASE 2014) (2014, September)

We present CoCoTest, a tool for automated testing of continuous controllers at the Model-in-the-Loop stage. CoCoTest combines explorative and exploitative search algorithms to identify scenar- ios in the ... [more ▼]

We present CoCoTest, a tool for automated testing of continuous controllers at the Model-in-the-Loop stage. CoCoTest combines explorative and exploitative search algorithms to identify scenar- ios in the controller input space that violate or are likely to violate the controller requirements. This enables a scalable and systematic way to test continuous properties of such controllers. Our experi- ments show that CoCoTest identifies critical flaws in the controller design that are rarely found by manual testing and go unnoticed until late stages of embedded software system development. [less ▲]

Detailed reference viewed: 206 (31 UL)
Full Text
Peer Reviewed
See detailIdentifying Optimal Trade-Offs between CPU Time Usage and Temporal Constraints Using Search
Nejati, Shiva UL; Briand, Lionel UL

in International Symposium on Software Testing and Analysis (ISSTA 2014) (2014, July)

Integration of software from different sources is a critical activity in many embedded systems across most industry sectors. Software integrators are responsible for producing reliable systems that ... [more ▼]

Integration of software from different sources is a critical activity in many embedded systems across most industry sectors. Software integrators are responsible for producing reliable systems that fulfill various functional and performance requirements. In many situations, these requirements inversely impact one another. In particular, embedded system integrators often need to make compromises regarding some of the functional system properties to optimize the use of various resources, such as CPU time. In this paper, motivated by challenges faced by industry, we introduce a multi-objective decision support approach to help balance the minimization of CPU time usage and the satisfaction of temporal constraints in automotive systems. We develop a multi-objective, search-based optimization algorithm, specifically designed to work for large search spaces, to identify optimal trade-off solutions fulfilling these two objectives. We evaluated our algorithm by applying it to a large automotive system. Our results show that our algorithm can find solutions that are very close to the estimated ideal optimal values, and further, it finds significantly better solutions than a random strategy while being faster. Finally, our approach efficiently identifies a large number of diverse solutions, helping domain experts and other stakeholders negotiate the solutions to reach an agreement. [less ▲]

Detailed reference viewed: 224 (30 UL)
Full Text
Peer Reviewed
See detailArchitecture-Level Configuration of Large-Scale Embedded Software Systems
Behjati, Razieh; Nejati, Shiva UL; Briand, Lionel UL

in ACM Transactions on Software Engineering & Methodology (2014), 23(3),

Configuration in the domain of integrated control systems (ICS) is largely manual, laborious, and error-prone. In this paper, we propose a model-based configuration approach that provides automation ... [more ▼]

Configuration in the domain of integrated control systems (ICS) is largely manual, laborious, and error-prone. In this paper, we propose a model-based configuration approach that provides automation support for reducing configuration effort and the likelihood of configuration errors in the ICS domain. We ground our approach on componentbased specifications of ICS families. We then develop a configuration algorithm using constraint satisfaction techniques over finite domains to generate products that are consistent with respect to their ICS family specifications. We reason about the termination and consistency of our configuration algorithm analytically. We evaluate the effectiveness of our configuration approach by applying it to a real subsea oil production system. Specifically, we have rebuilt a number of existing verified product configurations of our industry partner. Our experience shows that our approach can automatically infer up to 50% of the configuration decisions, and reduces the complexity of making configuration decisions. [less ▲]

Detailed reference viewed: 243 (35 UL)
Full Text
Peer Reviewed
See detailTraceability and SysML Design Slices to Support Safety Inspections: A Controlled Experiment
Briand, Lionel UL; Falessi, Davide; Nejati, Shiva UL et al

in ACM Transactions on Software Engineering & Methodology (2014), 23(1),

Detailed reference viewed: 226 (49 UL)
Full Text
Peer Reviewed
See detailAutomated Model-in-the-Loop Testing of Continuous Controllers using Search
Matinnejad, Reza UL; Nejati, Shiva UL; Briand, Lionel UL et al

in 5th Symposium on Search-Based Software Engineering (SSBSE 2013), Springer Lecture Notes in Computer Science (2013, August)

The number and the complexity of software components embedded in today’s vehicles is rapidly increasing. A large group of these components monitor and control the operating conditions of physical devices ... [more ▼]

The number and the complexity of software components embedded in today’s vehicles is rapidly increasing. A large group of these components monitor and control the operating conditions of physical devices (e.g., components controlling engines, brakes, and airbags). These controllers are known as continuous controllers. In this paper, we study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed feedback loop system.We identify a set of common requirements characterizing the desired behavior of continuous controllers, and develop a search-based technique to automatically generate test cases for these requirements. We evaluated our approach by applying it to a real automotive air compressor module. Our experience shows that our approach automatically generates several test cases for which the MiL level simulations indicate potential violations of the system requirements. Further, not only do our approach generates better test cases faster than random test case generation, but we also achieve better results than test scenarios devised by domain experts. [less ▲]

Detailed reference viewed: 390 (82 UL)
Full Text
Peer Reviewed
See detailMinimizing CPU Time Shortage Risks in Integrated Embedded Software
Nejati, Shiva UL; Adedjouma, Morayo UL; Briand, Lionel UL et al

in 28th IEEE/ACM International Conference on Automated Software Engineering (2013)

Detailed reference viewed: 276 (31 UL)