A Machine Learning Approach for Developing Test Oracles for Testing Scientific Software

Abstract

Abstract-Absence of test oracles is the grand challenge for testing complex scientific software. Metamorphic testing is the novel technique for developing test oracles on metamorphic relations. Although it is easy to find metamorphic relations based on general guidelines and domain knowledge, the ones that can adequately test the software are difficult to be developed. This paper introduces a machine learning approach for iteratively developing metamorphic relations. The approach develops initial metamorphic relations and tests first, and then the relations and tests are refined through mining the initial test execution and evaluation results with machine learning algorithms. The approach and its effectiveness are illustrated through testing an open source discrete dipole approximation program.

The Integration of Machine Learning into Automated Test Generation: A Systematic Literature Review

“…[96] model a function that judges whether an integer is prime-a binary classification problem. [85,90,91,92,93] also generate oracles for applications with enumerated output.…”

Using machine learning to generate test oracles: a systematic literature review

“…Shahamiri et al [32,34] generate oracles for a car insurance application, while Singhal et al [35] and Vanmali et al [37] generate oracles for a credit analysis at a bank. Ding et al [6] generate oracles for an image processing function that classifies a type of cell from image sections. All of these applications produce output from an enumerated set of values, easing the difficulty of generating an oracle.…”

“…Ding et al [6] used a support vector machine (SVM) to perform label propagation. Label propagation is a semi-supervised learning technique, where a mixture of labeled and unlabeled training data is used to train the model, and the algorithm attempts to propagate labels from the labelled data to similar, unlabeled data.…”

“…In terms of evaluation, five of the studies are focused on the accuracy of the oracle in a set of cases where the ground truth is known-measuring the percentage of correct classifications [1,16,31,41] or the accuracy of the model [35]. The remaining seven used the mutation score as the evaluation metric [6,24,32,34,[37][38][39].…”

Challenges for Verifying and Validating Scientific Software in Computational Materials Science

“…Identifying such relations is highly domain dependent. However, automatic techniques based on machine learning have been proposed [44], [50]- [52] and successfully applied to the bio-medical [53] and particle physics [54] domains. Transferring the concept of metamorphic testing requires domain expertise since the identified relations need to be understood and explained.…”