A Survey on Metamorphic Testing

Abstract

Abstract-A test oracle determines whether a test execution reveals a fault, often by comparing the observed program output to the expected output. This is not always practical, for example when a program's input-output relation is complex and difficult to capture formally. Metamorphic testing provides an alternative, where correctness is not determined by checking an individual concrete output, but by applying a transformation to a test input and observing how the program output "morphs" into a different one as a result. Since the introduction of such metamorphic relations in 1998, many contributions on metamorphic testing have been made, and the technique has seen successful applications in a variety of domains, ranging from web services to computer graphics. This article provides a comprehensive survey on metamorphic testing: It summarises the research results and application areas, and analyses common practice in empirical studies of metamorphic testing as well as the main open challenges.

Metamorphic Testing of Navigation Software: A Pilot Study with Google Maps

“…MT [5], [8], [9] alleviates the oracle problem by testing the SUT against prescribed MRs, which are necessary properties of the intended program's behavior. The difference between MRs and other types of program correctness properties is that an MR involves multiple executions of the target program.…”

“…A growing body of research and industrial application has investigated the concept of metamorphic testing (MT) [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], and has proven MT to be a highly effective testing paradigm for the detection of real-life software faults in the absence of an ideal test oracle. The idea of MT is simple: Instead of focusing on the correctness of each individual output, MT looks at the relationships among the inputs and outputs of multiple executions of the SUT.…”

“…Although the basic concept of MT is simple, it requires specific study when applied to different application domains [9]. This is because different application areas can have different properties of interest to investigate.…”

Towards Ex Vivo Testing of MapReduce Applications

“…This kind of oracle can detect some faults without any knowledge about the expected output. The oracle employed in MrExist is automatically derived from the program executions [71] using metamorphic testing [72]- [74], that is a field also employed to test machine learning programs [75] and in In Vivo frameworks [76] In most metamorphic testing research, the test cases are generated with random testing [77]. In MrExist, the original test cases are also obtained randomly based on a sampling of the production dataset.…”

“…According to the study of Segura et al [77] the number of metamorphic papers will increase in the following years, but to date 49% employ the metamorphic testing capabilities to different problem domains, and only 2% present a tool. In our case, this paper not only defines and automatizes the metamorphic relationship to the MapReduce domain, but also develops a tool that detects faults in production without human intervention and non-intrusively from runtime data.…”