Boosting Exploratory Testing of Industrial Automation Systems with AI

Eidenbenz, Raphael; Franke, Carsten; Sivanthi, Thanikesavan; Schoenborn, Sandro

doi:10.1109/icst49551.2021.00048

Cited by 4 publications

(7 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Almost all of the proposed approaches utilize a single ML technique. An approach explored in many domains is an ensemble [36]. In such approaches, models are trained on the same data using a variety of techniques.…”

Section: Resultsmentioning

confidence: 99%

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

Machine learning (ML) may enable effective automated test generation. We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges in this intersection by performing. We perform a systematic mapping study on a sample of 124 publications. ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property‐based, and expected output oracles. Supervised learning—often based on neural networks—and reinforcement learning—often based on Q‐learning—are common, and some publications also employ unsupervised or semi‐supervised learning. (Semi‐/Un‐)Supervised approaches are evaluated using both traditional testing metrics and ML‐related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. The work‐to‐date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed—and how they are applied—benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

Section: Resultsmentioning

confidence: 99%

The integration of machine learning into automated test generation: A systematic mapping study

Fontes

Gay

2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…[30] train a NN to identify usage behaviors likely to lead to failures using existing test cases. [27] randomly-generate a set of inputs and label them on whether they failed, then cluster failing instances to enhance accuracy.…”

Section: Examining Specific Practicesmentioning

confidence: 99%

“…Almost all of the proposed approaches utilize a single ML technique. An approach explored in many domains is the use of ensembles [27]. In such approaches, models are trained on the same data using a variety of techniques.…”

Section: Rq6: Limitations and Open Challengesmentioning

confidence: 99%

See 1 more Smart Citation

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Fontes¹,

Gay²

2022

Preprint

View full text Add to dashboard Cite

Context: Machine learning (ML) may enable effective automated test generation.Objectives: We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges.Methods: We perform a systematic literature review on a sample of 97 publications.Results: ML generates input for system, GUI, unit, performance, and combinatorial testing or improves the performance of existing generation methods. ML is also used to generate test verdicts, property-based, and expected output oracles. Supervised learning-often based on neural networks-and reinforcement learning-often based on Q-learning-are common, and some publications also employ unsupervised or semi-supervised learning. (Semi-/Un-)Supervised approaches are evaluated using both traditional testing metrics and ML-related metrics (e.g., accuracy), while reinforcement learning is often evaluated using testing metrics tied to the reward function. Conclusion:Work-to-date shows great promise, but there are open challenges regarding training data, retraining, scalability, evaluation complexity, ML algorithms employed-and how they are applied-benchmarks, and replicability. Our findings can serve as a roadmap and inspiration for researchers in this field.

show abstract

“…On the automation of ET, Eidenbenz et al (2016) employed arti cial intelligence techniques to predict test cases that are likely to cause failure in testing an industry control so ware. Makondo et al (2016) used neural networks to train test oracles to help the analysis of test results.…”

Section: Exploratory Testingmentioning

confidence: 99%

Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing

Zhu¹,

Bayley²

2021

Preprint

View full text Add to dashboard Cite

Testing has been widely recognised as di cult for AI applications. is paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. In these strategies, testing aims at exploring the data space of a classi cation or clustering application to discover the boundaries between classes that the machine learning application de nes.is enables the tester to understand precisely the behaviour and function of the so ware under test. In the paper, three variants of exploratory strategies are presented with the algorithms implemented in the automated datamorphic testing tool Morphy. e correctness of these algorithms are formally proved. eir capability and cost of discovering borders between classes are evaluated via a set of controlled experiments with manually designed subjects and a set of case studies with real machine learning models.

show abstract

Boosting Exploratory Testing of Industrial Automation Systems with AI

Cited by 4 publications

References 6 publications

The integration of machine learning into automated test generation: A systematic mapping study

The integration of machine learning into automated test generation: A systematic mapping study

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing

Contact Info

Product

Resources

About