PAT: A pattern classification approach to automatic reference oracles for the testing of mesh simplification programs

Abstract

Graphics applications often need to manipulate numerous graphical objects stored as polygonal models. Mesh simplification is an approach to vary the levels of visual details as appropriate, thereby improving on the overall performance of the applications. Different mesh simplification algorithms may cater for different needs, producing diversified types of simplified polygonal model as a result. Testing mesh simplification implementations is essential to assure the quality of the graphics applications. However, it is very difficult to determine the oracles (or expected outcomes) of mesh simplification for the verification of test results.A reference model is an implementation closely related to the program under test. Is it possible to use such reference models as pseudo-oracles for testing mesh simplification programs? If so, how effective are they?This paper presents a fault-based pattern classification methodology, called PAT, to address the questions. In PAT, we train the C4.5 classifier using black-box features of samples from a reference model and its fault-based versions, in order to test samples from the subject program. We evaluate PAT using four implementations of mesh simplification algorithms as reference models applied to 44 open-source three-dimensional polygonal models. Empirical results reveal that the use of a reference model as a pseudo-oracle is effective for testing the implementations of resembling mesh simplification algorithms. However, the results also show a tradeoff: When compared with a simple reference model, the use of a resembling but sophisticated reference model is more effective and accurate but less robust.

Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking

“…They then used the classifiers as test oracles. Chan et al [5] further investigated the feasibility of using pattern classification techniques when the test outputs cannot be accurately determined. Podgurski and colleagues [32][33] classified failure reports into categories via classifiers, and then refined the classification with the aim to extract more information to help testers diagnose program failure s. Bowring et al [34] used a progressive approach to train a classifier to ease the test oracle problem in regression testing.…”

“…The oracle problem is not limited to the above kind of scenarios. For instance, for programs involving complex computations (such as partial differential equations [4], graphics-based software [5][6], database applications [7], large-scale components, web server, or operating systems [7]), their outputs are difficult to verify.…”

“…Chan and others [11][18] also developed a metamorphic approach to online testing of service-oriented software applications. The improvement on the binary classification approach to alleviate the test oracle problem for graphics-intensive applications has been investigated in [5] [6]. Throughout these studies, both the testing and the evaluation of experimental results were conducted by the researchers themselves.…”

“…The subjects were required to apply both MT and assertion checking strategies to test these programs independently. We ran their test cases over a representative set of faulty versions of the target programs to assess the capability of these two testing strategies in detecting faults [5] [20]. The raw data were analyzed with a view to comparing the costs and effectiveness between MT and assertion checking.…”

A mapping study on testing non-testable systems

“…The SUT can be executed with a test case, and this trained machine learning algorithm can then be used to check for such patterns in this test case execution. For example, Chan et al (2009) constructed a training dataset, in which each data item corresponded to an individual test case, and consisted of a set of features that characterised the input and output of this test case. Each data item was also marked as "passed" or "failed".…”