Constructing Automated Test Oracle for Low Observable Software

Abstract

The application of machine learning techniques for constructing automated test oracles has been successful in recent years. However, existing machine learning based oracles are characterized by a number of de ciencies when applied to software systems with low observability, such as embedded software, cyber-physical systems, multimedia software programs, and computer games. This paper proposes a new black box approach to construct automated oracles that can be applied to software systems with low observability. The proposed approach employs an Arti cial Neural Network algorithm that uses input values and corresponding pass/fail outcomes of the program under test as the training set. To evaluate the performance of the proposed approach, extensive experiments were carried out on several benchmarks. The results manifest the applicability of the proposed approach to software systems with low observability and its higher accuracy than a well-known machine learning based method. This study also assessed the e ect of di erent parameters on the accuracy of the proposed approach.

A Literature Survey of Assertions in Software Testing

“…DSpot [18] takes developer-written test cases as input and synthesizes improved versions of them by triggering new behaviors and adding new assertions. Valueian et al [79] employ an Artificial Neural Network to construct automated oracles for low observable software based on tests inputs and verdict. Abdi et al [1] address test amplification for dynamically typed languages (e.g., Pharo), and exploit profiling information to infer the necessary type information creating special test inputs with corresponding assertions.…”

“…GASSERT [74] applies a co-evolutionary algorithm that explores the space of possible assertions to improve test oracles. Valueian et al [79] employ an Neural Network algorithm to construct automated test oracles for low observable software. A3Test [2] uses a pre-trained language model of assertions to generate assertions in test case generation process.…”

“…While the most common language is Java (e.g., [86] [34] [33] [74] [52]), a variety of other general-/specific-purpose languages have been covered, including JavaScript/TypeScript [91], Go [47], and Pharo Smalltalk [1]. Other solutions (e.g., [79] [49]) are not limited to a specific programming language and are applicable to programs in different languages. For example, Valueian et al [79] demonstrate the application of their solution on programs in Java, C, C++, Verilog, and VHDL.…”

Automated Software Test Data Generation With Generative Adversarial Networks

“…However, since the expected test outcomes should be manually generated, it is the most costly process in software testing. There are several types of research results for the generation of the expected output with the artificial neural network (ANN) [4]- [7]. Valueian et al [4] proposed a classifier-based method using ANNs that can build automated oracles for embedded software that has low observability and/or produces unstructured or semi-structured outputs.…”

“…There are several types of research results for the generation of the expected output with the artificial neural network (ANN) [4]- [7]. Valueian et al [4] proposed a classifier-based method using ANNs that can build automated oracles for embedded software that has low observability and/or produces unstructured or semi-structured outputs. In the proposed approach, the oracles need input data tagged with two labels of "pass" and "fail" rather than outputs and any execution trace.…”