Using machine learning techniques to detect metamorphic relations for programs without test oracles

Kanewala, Upulee; Bieman, James M.

doi:10.1109/issre.2013.6698899

Cited by 80 publications

(79 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Metamorphic Property in conjunction with the MTG is called a Metamorphic Relation (MR). MRs are evaluated by executing the MTG and checking that the Metamorphic Property holds (Kanewala and Bieman 2013b) between these executions; in this case checking that the price of SC 1 is greater than the price of SC 2 , and the price of SC 1 is greater than the price of SC 3 .…”

Section: Metamorphic Testingmentioning

confidence: 99%

“…However, the technique has an important limitation; it can currently only support MRs that are composed of one source and follow up test case (Chen et al 2016). Kanewala and Bieman (2013b) alternatively propose training Machine Learning classifiers to recognise operation sequence patterns that are correlated with particular MRs. Such a classifier can predict whether unseen code exhibits a particular MR.…”

Section: Effortmentioning

confidence: 99%

“…Such a classifier can predict whether unseen code exhibits a particular MR. Results have been promising; the technique has a low false positive rate, and can identify MRs even when faults are present in the SUT (Kanewala and Bieman 2013b).…”

Section: Effortmentioning

confidence: 99%

“…One may wish to extend the classifier with a graph kernel, 5 that has parameters ) that might have to be tuned to improve accuracy. Furthermore, since each classifier is associated with one MR type (Kanewala and Bieman 2013b), these additional tasks must be repeated for each MR type.…”

Section: Effortmentioning

confidence: 99%

See 3 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

The terms "Oracle Problem" and "Non-testable system" interchangeably refer to programs in which the application of test oracles is infeasible. Test oracles are an integral part of conventional testing techniques; thus, such techniques are inoperable in these programs. The prevalence of the oracle problem has inspired the research community to develop several automated testing techniques that can detect functional software faults in such programs. These techniques include N-Version testing, Metamorphic Testing, Assertions, Machine Learning Oracles, and Statistical Hypothesis Testing. This paper presents a Mapping Study that covers these techniques. The Mapping Study presents a series of discussions about each technique, from different perspectives, e.g. effectiveness, efficiency, and usability. It also presents a comparative analysis of these techniques in terms of these perspectives. Finally, potential research opportunities within the non-testable systems problem domain are highlighted within the Mapping Study. We believe that the aforementioned discussions and comparative analysis will be invaluable for new researchers that are attempting to familiarise themselves with the field, and be a useful resource for practitioners that are in the process of selecting an appropriate technique for their context, or deciding how to apply their selected technique. We also believe that our own insights, which are embedded throughout these discussions and the comparative analysis, will be useful for researchers that are already accustomed to the field. It is our hope that the potential research opportunities that have been highlighted by the Mapping Study will steer the direction of future research endeavours.

show abstract

Section: Metamorphic Testingmentioning

confidence: 99%

Section: Effortmentioning

confidence: 99%

Section: Effortmentioning

confidence: 99%

Section: Effortmentioning

confidence: 99%

See 2 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

show abstract

“…To facilitate this labor-intensive work, Kanewala and Bieman [33] proposed a machine-learning based technique to predict whether a program contains some forms of MRs. For example, for the sin program, this technique predicts whether it contains a form of MR like "sin(x) − c1sin(x + c2) = 0" where c1 and c2 are constants, but does not generate the values of constants c1 and c2. Different from their technique, in this paper, we aim to automatically infer specific MRs instead.…”

Section: Introductionmentioning

confidence: 99%

Search-based inference of polynomial metamorphic relations

Zhang

Chen

Hao

et al. 2014

Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering

102

View full text Add to dashboard Cite

Metamorphic testing (MT) is an effective methodology for testing those so-called "non-testable" programs (e.g., scientific programs), where it is sometimes very difficult for testers to know whether the outputs are correct. In metamorphic testing, metamorphic relations (MRs) (which specify how particular changes to the input of the program under test would change the output) play an essential role. However, testers may typically have to obtain MRs manually.In this paper, we propose a search-based approach to automatic inference of polynomial MRs for a program under test. In particular, we use a set of parameters to represent a particular class of MRs, which we refer to as polynomial MRs, and turn the problem of inferring MRs into a problem of searching for suitable values of the parameters. We then dynamically analyze multiple executions of the program, and use particle swarm optimization to solve the search problem.To improve the quality of inferred MRs, we further use MR filtering to remove some inferred MRs.We also conducted three empirical studies to evaluate our approach using four scientific libraries (including 189 scientific functions). From our empirical results, our approach is able to infer many high-quality MRs in acceptable time (i.e., from 9.87 seconds to 1231.16 seconds), which are effective in detecting faults with no false detection.

show abstract

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Gartziandia

Arrieta

Ayerdi

et al. 2022

J Software Evolu Process

View full text Add to dashboard Cite

The software of systems of elevators needs constant maintenance to deal with new functionality, bug fixes, or legislation changes. To automatically validate the software of these systems, a typical approach in industry is to use regression oracles, which execute test inputs both in the software version under test and in a previous software version. However, these practices require a long test execution time and cannot be re‐used at different test phases. To deal with these issues, we propose Dispatching AlgoRIthm Oracle (DARIO), a test oracle that relies on regression machine‐learning algorithms to detect both functional and non‐functional problems of the system. The machine‐learning algorithms of this oracle are trained by using data from previously tested versions to predict reference functional and non‐functional performance values of the new versions. An empirical evaluation with an industrial case study demonstrates the feasibility of using our approach. A total of five regression learning algorithms were validated by using mutation testing techniques. For the context of functional bugs, the accuracy when predicting verdicts by DARIO ranged between 95% and 98%, across the different scenarios proposed. For the context of non‐functional bugs, were competitive too, having an accuracy when predicting verdicts by DARIO ranged between 83% and 87%.

show abstract

Using machine learning techniques to detect metamorphic relations for programs without test oracles

Cited by 80 publications

References 33 publications

A mapping study on testing non-testable systems

A mapping study on testing non-testable systems

Search-based inference of polynomial metamorphic relations

Machine learning‐based test oracles for performance testing of cyber‐physical systems: An industrial case study on elevators dispatching algorithms

Contact Info

Product

Resources

About