An Empirical Study on the Selection of Good Metamorphic Relations

Mayer, Johannes; Guderlei, Ralph

doi:10.1109/compsac.2006.24

Cited by 71 publications

(43 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Let MR 1 and MR 2 be two MRs, such that MR 1 is theoretically stronger than MR 2 . This means that if one can confirm that MR 1 holds with respect to the entire input domain, then this implies that MR 2 also holds with respect to the entire input domain (Mayer and Guderlei 2006). This implies that MR 1 can detect all of the faults that can be detected by MR 2 , in addition to other faults.…”

Section: Characteristicsmentioning

confidence: 76%

“…Proponents of this argument view an understanding of the algorithm structure as necessary (Chen et al 2004a). Although this argument has a strong theoretical foundation, Mayer and Guderlei (2006) have questioned the practicality of the position. One must consider all input-output pairs to deduce the relative strength of one MR to another, which can be infeasible in practice (Mayer and Guderlei 2006).…”

Section: Characteristicsmentioning

confidence: 99%

“…an MR that reflects the behaviours of the SUT to a greater degree of completeness and accuracy) can be highly effective (Mayer and Guderlei 2006 The fourth widely reported characteristic is strength. Let MR 1 and MR 2 be two MRs, such that MR 1 is theoretically stronger than MR 2 .…”

Section: Characteristicsmentioning

confidence: 99%

“…This suggests that strong MRs are not necessarily more effective than weak MRs (Chen et al 2004a), and weak MRs are therefore not redundant. Mayer and Guderlei (2006) realised that weak MRs can have more failure revealing test cases than stronger MRs. This may explain why weak MRs can be more effective.…”

Section: Characteristicsmentioning

confidence: 99%

“…For example, white-box analysis techniques (Dong et al 2013), or coverage information generated by regression testing (Cao et al 2013) could assist in the identification of MRs that have MTGs with different test cases. MRs that use a similar strategy to the SUT tend to have MTGs that have similar source and follow up test cases (Mayer and Guderlei 2006), and thus should be avoided. Different MRs can have different MTG sizes (Cao et al 2013).…”

Section: Coveragementioning

confidence: 99%

See 4 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

The terms "Oracle Problem" and "Non-testable system" interchangeably refer to programs in which the application of test oracles is infeasible. Test oracles are an integral part of conventional testing techniques; thus, such techniques are inoperable in these programs. The prevalence of the oracle problem has inspired the research community to develop several automated testing techniques that can detect functional software faults in such programs. These techniques include N-Version testing, Metamorphic Testing, Assertions, Machine Learning Oracles, and Statistical Hypothesis Testing. This paper presents a Mapping Study that covers these techniques. The Mapping Study presents a series of discussions about each technique, from different perspectives, e.g. effectiveness, efficiency, and usability. It also presents a comparative analysis of these techniques in terms of these perspectives. Finally, potential research opportunities within the non-testable systems problem domain are highlighted within the Mapping Study. We believe that the aforementioned discussions and comparative analysis will be invaluable for new researchers that are attempting to familiarise themselves with the field, and be a useful resource for practitioners that are in the process of selecting an appropriate technique for their context, or deciding how to apply their selected technique. We also believe that our own insights, which are embedded throughout these discussions and the comparative analysis, will be useful for researchers that are already accustomed to the field. It is our hope that the potential research opportunities that have been highlighted by the Mapping Study will steer the direction of future research endeavours.

show abstract

Section: Characteristicsmentioning

confidence: 76%

Section: Characteristicsmentioning

confidence: 99%

Section: Characteristicsmentioning

confidence: 99%

Section: Characteristicsmentioning

confidence: 99%

Section: Coveragementioning

confidence: 99%

See 3 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

show abstract

Automated metamorphic testing of variability analysis tools

Segura

Durán

Sánchez

et al. 2015

Software Testing Verif & Rel

View full text Add to dashboard Cite

SUMMARYVariability determines the capability of software applications to be configured and customized. A common need during the development of variability-intensive systems is the automated analysis of their underlying variability models, e.g. detecting contradictory configuration options. The analysis operations that are performed on variability models are often very complex, which hinders the testing of the corresponding analysis tools and makes difficult, often infeasible, to determine the correctness of their outputs, i.e. the well-known oracle problem in software testing. In this article, we present a generic approach for the automated detection of faults in variability analysis tools overcoming the oracle problem. Our work enables the generation of random variability models together with the exact set of valid configurations represented by these models. These test data are generated from scratch using step-wise transformations and assuring that certain constraints (a.k.a. metamorphic relations) hold at each step. To show the feasibility and generalizability of our approach, it has been used to automatically test several analysis tools in three variability domains: feature models, CUDF documents and Boolean formulas. Among other results, we detected 19 real bugs in 7 out of the 15 tools under test.

show abstract

Metamorphic relation prioritization for effective regression testing

Srinivasan

Kanewala

2022

Software Testing Verif & Rel

View full text Add to dashboard Cite

Metamorphic testing (MT) is widely used for testing programs that face the oracle problem. It uses a set of metamorphic relations (MRs), which are relations among multiple inputs and their corresponding outputs to determine whether the program under test is faulty. Typically, MRs vary in their ability to detect faults in the program under test, and some MRs tend to detect the same set of faults. In this paper, we propose approaches to prioritize MRs to improve the efficiency and effectiveness of MT for regression testing. We present two MR prioritization approaches: (i) fault-based and (ii) coverage-based. To evaluate these MR prioritization approaches, we conduct experiments on three complex open-source software systems. Our results show that the MR prioritization approaches developed by us significantly outperform the current practice of executing the source and follow-up test cases of the MRs in an adhoc manner in terms of fault detection effectiveness. Further, fault-based MR prioritization leads to reducing the number of source and follow-up test cases that needs to be executed as well as reducing the average time taken to detect a fault, which would result in saving time and cost during the testing process.

show abstract

An Empirical Study on the Selection of Good Metamorphic Relations

Cited by 71 publications

References 15 publications

A mapping study on testing non-testable systems

A mapping study on testing non-testable systems

Automated metamorphic testing of variability analysis tools

Metamorphic relation prioritization for effective regression testing

Contact Info

Product

Resources

About