Metamorphic Testing for Quality Assurance of Protein Function Prediction Tools

Shahri, Morteza Pourreza; Srinivasan, Madhusudan; Reynolds, Gillian; Bimczok, Diane; Kahanda, Indika; Kanewala, Upulee

doi:10.1109/aitest.2019.00017

Cited by 6 publications

(3 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Metamorphic Testing (MT) has been applied to a variety of domains and applications, see [24,2] for an in-depth overview. Successful application domains include testing of driverless cars [7], search engines [8], machine translation systems [25], performance testing [26,27], constraint solvers [28] or bioinformatics [9]. Previous works already focused on its automation, for example by exploring algorithms to specifically identify fault-revealing inputs [29] or performing an empirical study on selecting good Metamorphic Relations (MRs) [30].…”

Section: Related Workmentioning

confidence: 99%

“…MT has been very successful to address testing issues in various application domains, e.g., driverless cars [7], search engines [8], or bioinformatics [9] just to name a few (see Section 2.3 for more references). However, generally speaking, applying MT in practice requires to address two issues: the MR identification and the MR selection problems [2].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Adaptive metamorphic testing with contextual bandits

Spieker

Gotlieb

2020

Journal of Systems and Software

View full text Add to dashboard Cite

Metamorphic Testing is a software testing paradigm which aims at using necessary properties of a systemunder-test, called metamorphic relations, to either check its expected outputs, or to generate new test cases. Metamorphic Testing has been successful to test programs for which a full oracle is not available or to test programs for which there are uncertainties on expected outputs such as learning systems. In this article, we propose Adaptive Metamorphic Testing as a generalization of a simple yet powerful reinforcement learning technique, namely contextual bandits, to select one of the multiple metamorphic relations available for a program. By using contextual bandits, Adaptive Metamorphic Testing learns which metamorphic relations are likely to transform a source test case, such that it has higher chance to discover faults. We present experimental results over two major case studies in machine learning, namely image classification and object detection, and identify weaknesses and robustness boundaries. Adaptive Metamorphic Testing efficiently identifies weaknesses of the tested systems in context of the source test case.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Adaptive metamorphic testing with contextual bandits

Spieker

Gotlieb

2020

Journal of Systems and Software

View full text Add to dashboard Cite

show abstract

“…This test technique first generates a set of metamorphic relations (MRs) according to the nature of the test program and then tests the software by judging whether the test case satisfies the metamorphic relations. Since MT is proposed, it has been widely applied on the test of various programs in many fields for its simple and efficient characteristics, such as image processing [4,5], network diagnosis [6,7], machine learning [8,9], bioengineering [10], and scientific software [11]. However, the metamorphic relations are generally constructed manually by the testers or developers who understand the principle of the test program, or are determined according to prior knowledge, which is an obstacle to software test automation.…”

Section: Introductionmentioning

confidence: 99%

A Genetic Algorithm-Based Approach for Composite Metamorphic Relations Construction

Xiang

2019

Information

View full text Add to dashboard Cite

The test oracle problem exists widely in modern complex software testing, and metamorphic testing (MT) has become a promising testing technique to alleviate this problem. The inference of efficient metamorphic relations (MRs) is the core problem of metamorphic testing. Studies have proven that the combination of simple metamorphic relations can construct more efficient metamorphic relations. In most previous studies, metamorphic relations have been mainly manually inferred by experts with professional knowledge, which is an inefficient technique and hinders the application. In this paper, a genetic algorithm-based approach is proposed to construct composite metamorphic relations automatically for the program to be tested. We use a set of relation sequences to represent a particular class of MRs and turn the problem of inferring composite MRs into a problem of searching for suitable sequences. We then dynamically implement multiple executions of the program and use a genetic algorithm to search for the optimal set of relation sequences. We conducted empirical studies to evaluate our approach using scientific functions in the GNU scientific library (abbreviated as GSL). From the empirical results, our approach can automatically infer high-quality composite MRs, on average, five times more than basic MRs. More importantly, the inferred composite MRs can increase the fault detection capabilities by at least 30 % more than the original metamorphic relations.

show abstract

Metamorphic relation automation: Rationale, challenges, and solution directions

Altamimi

Elkawakjy

Çatal

2022

J Software Evolu Process

View full text Add to dashboard Cite

Metamorphic testing addresses the issue of the oracle problem by comparing results transformation from multiple test executions. The relationship that governs the output transformation is called metamorphic relation. Metamorphic relations require expert knowledge and the generation of them is considered a time-consuming task.Researchers have proposed various techniques to automate metamorphic testing, generation, and selection. Although there are several research articles on this issue, there is a lack of overview of the state-of-the-art of metamorphic relation automation. As such, we performed a systematic literature review study to collect, extract, and synthesize the required data. Based on our research questions, the literature was categorized and summarized into different categories. We found that the automation of metamorphic relation is most effective in mathematical and scientific applications.We concluded that some approaches involve analysis of different forms of softwarerelated information such as control flow graph and program dependence graph as well as an initial set of metamorphic relations. On the other hand, other methods involve analysis of executions of the software functions with random and specific inputs. The results show that this field is still in its infancy with opportunities for novel work, especially in methods utilizing machine learning.

show abstract

Metamorphic Testing for Quality Assurance of Protein Function Prediction Tools

Cited by 6 publications

References 37 publications

Adaptive metamorphic testing with contextual bandits

Adaptive metamorphic testing with contextual bandits

A Genetic Algorithm-Based Approach for Composite Metamorphic Relations Construction

Metamorphic relation automation: Rationale, challenges, and solution directions

Contact Info

Product

Resources

About