Statistical Metamorphic Testing Testing Programs with Random Output by Means of Statistical Hypothesis Tests and Metamorphic Testing

Guderlei, Ralph; Mayer, Johannes

doi:10.1109/qsic.2007.4385527

Cited by 51 publications

(52 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since MRs are domain specific (Chen et al 2009a), the total number of potential MRs in one problem domain can be different than in another. For example, Chen et al (2004a) found nine MRs for Dijkstra's Algorithm, whilst Guderlei and Mayer (2007a) could only find one MR for the inverse cumulative distribution function. Therefore, the problem domain is likely to directly influence MT's effectiveness.…”

Section: Problem Domainmentioning

confidence: 99%

“…SHT was developed to resolve this specific type of oracle problem (Guderlei and Mayer 2007a). SHT recognises that such systems may have a typical output distribution, and that information about this typical output distribution may be available prior to execution, even if information about the test outcome of a single execution is not.…”

Section: Statistical Hypothesis Testingmentioning

confidence: 99%

“…Another example of such an assumption is that the expected output distribution is known (Mayer 2005). One could use reference implementations (RIs) to determine the expected distribution (Guderlei and Mayer 2007a), if this assumption does not hold. Unfortunately, the negative issues that are associated with the use of RIs may also affect SHT, if this approach is used, e.g.…”

Section: Assumptionsmentioning

confidence: 99%

See 2 more Smart Citations

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

The terms "Oracle Problem" and "Non-testable system" interchangeably refer to programs in which the application of test oracles is infeasible. Test oracles are an integral part of conventional testing techniques; thus, such techniques are inoperable in these programs. The prevalence of the oracle problem has inspired the research community to develop several automated testing techniques that can detect functional software faults in such programs. These techniques include N-Version testing, Metamorphic Testing, Assertions, Machine Learning Oracles, and Statistical Hypothesis Testing. This paper presents a Mapping Study that covers these techniques. The Mapping Study presents a series of discussions about each technique, from different perspectives, e.g. effectiveness, efficiency, and usability. It also presents a comparative analysis of these techniques in terms of these perspectives. Finally, potential research opportunities within the non-testable systems problem domain are highlighted within the Mapping Study. We believe that the aforementioned discussions and comparative analysis will be invaluable for new researchers that are attempting to familiarise themselves with the field, and be a useful resource for practitioners that are in the process of selecting an appropriate technique for their context, or deciding how to apply their selected technique. We also believe that our own insights, which are embedded throughout these discussions and the comparative analysis, will be useful for researchers that are already accustomed to the field. It is our hope that the potential research opportunities that have been highlighted by the Mapping Study will steer the direction of future research endeavours.

show abstract

Section: Problem Domainmentioning

confidence: 99%

Section: Statistical Hypothesis Testingmentioning

confidence: 99%

Section: Assumptionsmentioning

confidence: 99%

See 1 more Smart Citation

A mapping study on testing non-testable systems

Patel

Hierons

2017

Software Qual J

View full text Add to dashboard Cite

show abstract

“…In the cases in which JSim uses non-deterministic event timing, we applied statistical metamorphic testing (SMT), which has been proposed as a technique for testing nondeterministic applications that do not have test oracles [16]. SMT can be applied to programs for which the output is numeric, such as the overall event timing in JSim, and is based on the statistical properties of multiple invocations of the program.…”

Section: Jsim Metamorphic Propertiesmentioning

confidence: 99%

“…In some cases, these works have looked at situations in which there cannot be an oracle for a particular application [12], as in the case of non-testable programs; in others, the work has considered the case in which the oracle is simply absent or difficult to implement [7]. Much of the work on applying metamorphic testing to applications without test oracles has focused on specific domains, such as machine learning [22], computational biology [10], or graphics [16]. To our knowledge, we are the first to focus on applying metamorphic testing to the domain of simulation software in general, and health care related simulation software in particular.…”

Section: Related Workmentioning

confidence: 99%

On effective testing of health care simulation software

Murphy

Raunak

King

et al. 2011

Proceedings of the 3rd Workshop on Software Engineering in Health Care

View full text Add to dashboard Cite

Health care professionals rely on software to simulate anatomical and physiological elements of the human body for purposes of training, prototyping, and decision making. Software can also be used to simulate medical processes and protocols to measure cost effectiveness and resource utilization. Whereas much of the software engineering research into simulation software focuses on validation (determining that the simulation accurately models real-world activity), to date there has been little investigation into the testing of simulation software itself, that is, the ability to effectively search for errors in the implementation. This is particularly challenging because often there is no test oracle to indicate whether the results of the simulation are correct. In this paper, we present an approach to systematically testing simulation software in the absence of test oracles, and evaluate the effectiveness of the technique.

show abstract

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Kanewala

Bieman

Ben‐Hur

2015

Software Testing Verif & Rel

View full text Add to dashboard Cite

Summary Comprehensive, automated software testing requires an oracle to check whether the output produced by a test case matches the expected behaviour of the programme. But the challenges in creating suitable oracles limit the ability to perform automated testing in some programmes, and especially in scientific software. Metamorphic testing is a method for automating the testing process for programmes without test oracles. This technique operates by checking whether the programme behaves according to properties called metamorphic relations. A metamorphic relation describes the change in output when the input is changed in a prescribed way. Unfortunately, finding the metamorphic relations satisfied by a programme or function remains a labour‐intensive task, which is generally performed by a domain expert or a programmer. In this work, we propose a machine learning approach for predicting metamorphic relations that uses a graph‐based representation of a programme to represent control flow and data dependency information. In earlier work, we found that simple features derived from such graphs provide good performance. An analysis of the features used in this earlier work led us to explore the effectiveness of several representations of those graphs using the machine learning framework of graph kernels, which provide various ways of measuring similarity between graphs. Our results show that a graph kernel that evaluates the contribution of all paths in the graph has the best accuracy and that control flow information is more useful than data dependency information. The data used in this study are available for download at http://www.cs.colostate.edu/saxs/MRpred/functions.tar.gz to help researchers in further development of metamorphic relation prediction methods. Copyright © 2015 John Wiley & Sons, Ltd.

show abstract

Statistical Metamorphic Testing Testing Programs with Random Output by Means of Statistical Hypothesis Tests and Metamorphic Testing

Cited by 51 publications

References 3 publications

A mapping study on testing non-testable systems

A mapping study on testing non-testable systems

On effective testing of health care simulation software

Predicting metamorphic relations for testing scientific software: a machine learning approach using graph kernels

Contact Info

Product

Resources

About