Learning-Based Testing for Safety Critical Automotive Applications

Machine learning (ML) may enable effective automated test generation. We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges in this intersection by performing. We perform a systematic mapping study on a sample of 124 publications. ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property‐based, and expected output oracles. Supervised learning—often based on neural networks—and reinforcement learning—often based on Q‐learning—are common, and some publications also employ unsupervised or semi‐supervised learning. (Semi‐/Un‐)Supervised approaches are evaluated using both traditional testing metrics and ML‐related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. The work‐to‐date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed—and how they are applied—benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

“…Khosrowjerdi et al combine supervised learning and model checking [107]. A model is learned from system executions that predicts output.…”

Section: Test Oracle Generationmentioning

confidence: 99%

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…[76] train a model to identify rendering errors in video games by training on screenshots of previous faults. [78] combine ML and model checking. A model is learned from system executions that predicts output.…”

Section: Test Oracle Generationmentioning

confidence: 99%

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

“…A great effort has been devoted to a more practical direction of BBC, including the testing of automotive systems. For example, case studies on testing of automotive software systems are shown in [32] and an application to the CPSs with continuous dynamics is presented in [31,39]. However, up to our knowledge, there is no work exploiting the quantitative satisfaction degree of the requirements in addition to Boolean satisfaction.…”

Section: Related Workmentioning

confidence: 99%

Falsification of cyber-physical systems with robustness-guided black-box checking

Waga

2020

Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control

View full text Add to dashboard Cite

For exhaustive formal verification, industrial-scale cyberphysical systems (CPSs) are often too large and complex, and lightweight alternatives (e.g., monitoring and testing) have attracted the attention of both industrial practitioners and academic researchers. Falsification is one popular testing method of CPSs utilizing stochastic optimization. In stateof-the-art falsification methods, the result of the previous falsification trials is discarded, and we always try to falsify without any prior knowledge. To concisely memorize such prior information on the CPS model and exploit it, we employ Black-box checking (BBC), which is a combination of automata learning and model checking. Moreover, we enhance BBC using the robust semantics of STL formulas, which is the essential gadget in falsification. Our experiment results suggest that our robustness-guided BBC outperforms a state-of-the-art falsification tool. CCS CONCEPTS• Computer systems organization → Embedded and cyber-physical systems; Real-time system specification; • Software and its engineering → Formal software verification; Search-based software engineering.

show abstract

Learning-Based Testing for Safety Critical Automotive Applications

Cited by 10 publications

References 16 publications

The integration of machine learning into automated test generation: A systematic mapping study

The integration of machine learning into automated test generation: A systematic mapping study

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Falsification of cyber-physical systems with robustness-guided black-box checking

Contact Info

Product

Resources

About