Metamorphic Testing: A Simple Yet Effective Approach for Testing Scientific Software

Kanewala, Upulee; Chen, Tsong Yueh

doi:10.1109/mcse.2018.2875368

Cited by 11 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Metamorphic testing is one of the effective means to solve Oracle problems (Chen et al, 1998;Liu et al, 2014;Segura et al, 2018;Kanewala and Yueh Chen 2019). Studies have shown that MT has the advantages of reasonable cost and a more vital ability to expose errors (Hu et al, 2006).…”

Section: Metamorphic Testingmentioning

confidence: 99%

A Lightweight Verification Method Based on Metamorphic Relation for Nuclear Power Software

Yang

Yan

et al. 2022

Front. Energy Res.

View full text Add to dashboard Cite

The verification of nuclear design software commonly uses direct comparison methods. Benchmark questions, classical programs, experimental data, manual solutions, etc., would be used as expected results to compare with program outputs to evaluate the reliability of software coding and the accuracy of the numerical solution. Because nuclear power software numerically simulates complex physical processes, it involves many partial differential equations. It is usually challenging to construct analytical or accurate solutions and is expensive to develop benchmark questions and experimental data. Hence, the quantity of verification examples is small. By using the direct comparison method, verification is complicated, high cost, and inadequate. Entering the validation process without adequate proof will adversely impact the effectiveness and efficiency of validation. Metamorphic testing is an indirect verification technology that cleverly combines the nature of the model with software verification. It evaluates the correctness of the code by examining whether the program satisfies the metamorphic relation. Without manual solutions or benchmark examples, it has broad application prospects in the field of nuclear power. A lightweight verification method based on metamorphic relation has been produced here. Metamorphic relations are identified from physical equations, numerical algorithms, and program specifications. Next, they are explicitly used to system, integration, and unit tests to improve test adequacy. Because no need to develop verification examples, this method can detect code errors as soon as possible at a low cost, improve test efficiency, avoid mistakes remaining in subsequent stages and reduce the overall cost of verification.

show abstract

Section: Metamorphic Testingmentioning

confidence: 99%

A Lightweight Verification Method Based on Metamorphic Relation for Nuclear Power Software

Yang

Yan

et al. 2022

Front. Energy Res.

View full text Add to dashboard Cite

show abstract

“…MRs have been used in traditional software programming to test scientific software [19], web services [35] and image software [17] among others [33]. Selection or generation of MRs applied to traditional software was also researched on such as using machine learning and graph-based representation relations to generate new ones [18] which restricts the application to program where such a graph can be used.…”

Section: Related Workmentioning

confidence: 99%

HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks

Tambon¹,

Antoniol²,

Khomh³

2021

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNN) applications are increasingly becoming a part of our everyday life, from medical applications to autonomous cars. Traditional validation of DNN relies on accuracy measures, however, the existence of adversarial examples has highlighted the limitations of these accuracy measures, raising concerns especially when DNN are integrated into safety-critical systems. In this paper, we present HOMRS, an approach to boost metamorphic testing by automatically building a small optimized set of high order metamorphic relations from an initial set of elementary metamorphic relations. HOMRS' backbone is a multi-objective search; it exploits ideas drawn from traditional systems testing such as code coverage, test case, and path diversity. We applied HOMRS to LeNet5 DNN with MNIST dataset and we report evidence that it builds a small but effective set of high order transformations achieving a 95% kill ratio. Five raters manually labelled a pool of images before and after high order transformation; Fleiss' Kappa and statistical tests confirmed that they are metamorphic properties. HOMRS built-in relations are also effective to confront adversarial or out-of-distribution examples; HOMRS detected 92% of randomly sampled out of distribution images. HOMRS transformations are also suitable for on-line real-time use. CCS Concepts: • Computing methodologies → Neural networks.

show abstract

“…For instances, in testing simulations, MT has been successfully adopted to test agent-based (ABM) and discrete-event (DES) simulations [26], hybrid ABM and DES systems [11], health care simulation [25], webenabled simulation [1], and simulator platform for self-driving cars [36,41]. In testing scientific software, MT is an effective technique to detect faults in simulation programs for designing nuclear power plants [14], bioinformatics programs [4], epidemiological models [31,33], chemical reaction networks for prototyping nano-scale molecular devices [13], matrix calculation programs [32], solvers for partial differential equations [3], multiple linear regression software [22], ocean modelling [16], storm water management model systems [19], machine learning-based hydro-logical models [40], Monte-Carlo computational programs [10,30], serverless scientific applications [20], as well as other types of scientific software [9,18,19,29]. For example, He et al [14] has found 33 bugs in simulation programs that are used to design and analyze nuclear power plants in a study that adopts MT.…”

Section: Introductionmentioning

confidence: 99%

Testing Ocean Software with Metamorphic Testing

Luu,

Liu,

Chen

et al. 2022

Preprint

View full text Add to dashboard Cite

Advancing ocean science has a significant impact to the development of the world, from operating a safe navigation for vessels to maintaining a healthy and diverse ocean ecosystem. Various ocean software systems have been extensively adopted for different purposes, for instance, predicting hourly sea level elevation across shorelines, simulating large-scale ocean circulations, as well as integrating into Earth system models for weather forecasts and climate projections. Regardless of their significance, guaranteeing the trustworthiness of ocean software and modelling systems is a long-standing challenge. The testing of ocean software suffers a lot from the so-called oracle problem, which refers to the absence of test oracles mainly due to the nonlinear interactions of multiple physical variables and the high complexity in computation. In the ocean, observed tidal signals are distorted by non-deterministic physical variables, hindering us from knowing the "true" astronomical tidal constituents existing in the timeseries. In this paper, we present how to test tidal analysis and prediction (TAP) software based on metamorphic testing (MT), a simple yet effective testing approach to the oracle problem. In particular, we construct metamorphic relations from the periodic property of astronomical tide, and then use them to successfully detect a real-life defect in an open-source TAP software. We also conduct a series of experiments to further demonstrate the applicability and effectiveness of MT in the testing of TAP software. Our study not only justifies the potential of MT in testing more complex ocean software and modelling systems, but also can be expanded to assess and improve the quality of a broader range of scientific simulation software systems. CCS CONCEPTS• Applied computing → Earth and atmospheric sciences; • Software and its engineering → Software testing and debugging.

show abstract

Metamorphic Testing: A Simple Yet Effective Approach for Testing Scientific Software

Cited by 11 publications

References 10 publications

A Lightweight Verification Method Based on Metamorphic Relation for Nuclear Power Software

A Lightweight Verification Method Based on Metamorphic Relation for Nuclear Power Software

HOMRS: High Order Metamorphic Relations Selector for Deep Neural Networks

Testing Ocean Software with Metamorphic Testing

Contact Info

Product

Resources

About