Combination and mutation strategies to support test data generation in the context of autonomous vehicles

Neves, Vânia de Oliveira; Delamaro, Márcio Eduardo; Masiero, Paulo César

doi:10.1504/ijes.2016.10001345

Cited by 2 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A further challenge is assessing whether the results obtained in a simulated environment would also scale to the real world (de Oliveira Neves et al 2016;Wolschke et al 2018;Li et al 2016). Two works propose to generate realistic test scenarios from in-field data (de Oliveira Neves et al 2016), or by mining test cases from real-world traffic situations or traffic simulators (Wolschke et al 2018).…”

Section: Addressed Problem (Rq 11)mentioning

confidence: 99%

“…7 Testing Levels i.e., in principle it may be applicable to any MLS (Aniculaesei et al 2018;Byun et al 2019;Cheng et al 2018a, b;Du et al 2019;Eniser et al 2019;Guo et al 2018;Henriksson et al 2019;Kim et al 2019;Li et al 2018;Ma et al 2018bMa et al , c, d, 2019Murphy et al 2007aMurphy et al , b, 2008Murphy et al , b, 2009Nakajima and Bui 2016, 2019Odena et al 2019;Pei et al 2017;Saha and Kanewala 2019;Sekhon and Fleming 2019;Shen et al 2018;Shi et al 2019;Sun et al 2018a, b;Tian et al 2018;Udeshi and Chattopadhyay 2019;Uesato et al 2019;Xie et al 2018Xie et al , 2019Xie et al , 2011Zhang et al 2018aZhang et al , 2019Zhao and Gao 2018). Around 30% proposed approaches are designed for autonomous systems (Abeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Patel et al 2018;Strickland et al 2018;Wolschke et al 2017;Fremont et al 2019), am...…”

Section: Domains (Rq 13)mentioning

confidence: 99%

“…Nearly 60% (33 out of 55) of the papers present various methods to generate test inputs for the MLS Fig. 10 Test Artefacts under test (Abdessalem et al 2016(Abdessalem et al , 2018aAbeysirigoonawardena et al 2019;Beglerovic et al 2017;Bühler and Wegener 2004;Du et al 2019;Eniser et al 2019;Fremont et al 2019;Guo et al 2018;Klueck et al 2018;Li et al 2016;Ma et al 2018cMa et al , 2019Majumdar et al 2019;Murphy et al 2007a, b;de Oliveira Neves et al 2016;Odena et al 2019;Pei et al 2017;Sekhon and Fleming 2019;Sun et al 2018a, b;Tian et al 2018;Tuncali et al 2018Tuncali et al , 2019Udeshi et al 2018Udeshi et al , 2019Wolschke et al 2017Wolschke et al , 2018Xie et al 2019;Zhang et al 2018b;Zheng et al 2018b). However, what a test input represents differs across the proposed approaches and highly depends on the domain of the tested system.…”

Section: Test Artefacts (Rq 21)mentioning

confidence: 99%

“…The proposed testing approaches require different levels of access to the MLS under test. In 29 cases out of 70 (41%), it is enough to have a black-box access to the system (Abdessalem et al 2016(Abdessalem et al , 2018aAbeysirigoonawardena et al 2019;Aniculaesei et al 2018;Beglerovic et al 2017;Bolte et al 2019;Bühler and Wegener 2004;Xie et al 2018;Fremont et al 2019;Klueck et al 2018;Li et al 2016;Majumdar et al 2019;Mullins et al 2018;Murphy et al , 2009Nakajima 2018;de Oliveira Neves et al 2016;Patel et al 2018;Qin et al 2018;Rubaiyat et al 2018;Tuncali et al 2018Tuncali et al , 2019Udeshi and Chattopadhyay 2019;Uesato et al 2019;Wolschke et al 2018;Wolschke et al 2017;Zhao and Gao 2018;Zheng et al 2019;Zhang et al 2016). In 11 cases (16%), along with the inputs and outputs of MLS, training data should also be available: black-box access is not sufficient and data-box access to the system should be provided (Cheng et al 2018b;Ding et al 2017;Henriksson et al 2019;Saha and Kanewala 2019;Udeshi et al 2018;Xie et al 2011;Dwarakanath et al 2018;Groce et al 2014;Zhang et al 2018b;Murphy et al 2007b;…”

Section: Access To the System (Rq 25)mentioning

confidence: 99%

“…In our pool of papers, 17 works (24%) model the context in which the MLS operates. All of them are in the autonomous driving domain (Abdessalem et al 2016, b, Abeysirigoonawardena et al 2019Beglerovic et al 2017;Bolte et al 2019;Bühler and Wegener 2004;Cheng et al 2018b;Fremont et al 2019;Klueck et al 2018;Li et al 2016;Mullins et al 2018;de Oliveira Neves et al 2016;Tuncali et al 2018Tuncali et al , 2019Wolschke et al 2017;Zhao and Gao 2018). The context models presented in these papers vary in terms of simplicity and number of considered actors.…”

Section: Context Model (Rq 26)mentioning

confidence: 99%

See 4 more Smart Citations

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

Context: A Machine Learning based System (MLS) is a software system including one or more components that learn how to perform a task from a given data set. The increasing adoption of MLSs in safety critical domains such as autonomous driving, healthcare, and finance has fostered much attention towards the quality assurance of such systems. Despite the advances in software testing, MLSs bring novel and unprecedented challenges, since their behaviour is defined jointly by the code that implements them and the data used for training them. Objective: To identify the existing solutions for functional testing of MLSs, and classify them from three different perspectives: (1) the context of the problem they address, (2) their features, and (3) their empirical evaluation. To report demographic information about the ongoing research. To identify open challenges for future research. Method: We conducted a systematic mapping study about testing techniques for MLSs driven by 33 research questions. We followed existing guidelines when defining our research protocol so as to increase the repeatability and reliability of our results. Results: We identified 70 relevant primary studies, mostly published in the last years. We identified 11 problems addressed in the literature. We investigated multiple aspects of the testing approaches, such as the used/proposed adequacy criteria, the algorithms for test input generation, and the test oracles. Conclusions: The most active research areas in MLS testing address automated scenario/input generation and test oracle creation. MLS testing is a rapidly growing and developing research area, with many open challenges, such as the generation of realistic inputs and the definition of reliable evaluation metrics and benchmarks.

show abstract

Section: Addressed Problem (Rq 11)mentioning

confidence: 99%

Section: Domains (Rq 13)mentioning

confidence: 99%

Section: Test Artefacts (Rq 21)mentioning

confidence: 99%

Section: Access To the System (Rq 25)mentioning

confidence: 99%

Section: Context Model (Rq 26)mentioning

confidence: 99%

See 3 more Smart Citations

Testing machine learning based systems: a systematic mapping

et al. 2020

View full text Add to dashboard Cite

show abstract

ExVivoMicroTest: ExVivo Testing of Microservices

Gazzola

Goldstein²,

Mariani

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

Microservice‐based applications consist of multiple services that can evolve independently. When a service must be updated, it is first tested with in‐house regression test suites. However, the test suites that are executed are usually designed without the exact knowledge about how the services will be accessed and used in the field; therefore, they may easily miss relevant test scenarios, failing to prevent the deployment of faulty services. To address this problem, we introduce ExVivoMicroTest, an approach that analyzes the execution of deployed services at run‐time in the field, in order to generate test cases for future versions of the same services. ExVivoMicroTest implements lightweight monitoring and tracing capabilities, to inexpensively record executions that can be later turned into regression test cases that capture how services are used in the field. To prevent accumulating an excessive number of test cases, ExVivoMicroTest uses a test coverage model that can discriminate the recorded executions between the ones that are worth to be turned into test cases and the ones that should be discarded. The resulting test cases use a mocked environment that fully isolates the service under test from the rest of the system to faithfully reply interactions. We assessed ExVivoMicroTest with the PiggyMetrics and Train Ticket open source microservice applications and studied how different configurations of the monitoring and tracing logic impact on the capability to generate test cases.

show abstract

Combination and mutation strategies to support test data generation in the context of autonomous vehicles

Cited by 2 publications

References 0 publications

Testing machine learning based systems: a systematic mapping

Testing machine learning based systems: a systematic mapping

ExVivoMicroTest: ExVivo Testing of Microservices

Contact Info

Product

Resources

About