Building Test Oracles by Clustering Failures

Almaghairbe, Rafig; Roper, Marc

doi:10.1109/ast.2015.8

Cited by 3 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since failure‐causing inputs often exhibit “clustering” effects, the unsupervised learning methods represented by cluster analysis are widely used to characterize the distribution of failure‐causing inputs 69‐71 . Based on the anomaly detection based on clustering, Almaghairbe et al 72 presented a study of building test oracles according to the hypothesis that failures will tend to congregate in small clusters. In addition, for multiple faults scenarios, the clustering technique is used to divide the failed test cases into several disjoint groups to help developers perform parallel debugging 73 …”

Section: Related Workmentioning

confidence: 99%

A lightweight adaptive random testing method for deep learning systems

Mao,

Song,

Chen

2023

Softw Pract Exp

View full text Add to dashboard Cite

In recent years, deep learning (DL) systems are increasingly used in the safety‐critical fields such as autonomous driving, medical diagnosis, and financial service. Although these systems have demonstrated an outstanding performance in enhancing the accuracy of decision‐making, they pose significant challenges to the trustworthiness due to their limited interpretability and inherent uncertainty. Adaptive random testing (ART) has been proved as an effective approach for ensuring the reliability of DL systems. However, existing ART methods for DL systems incur a heavy overhead in test case selection due to the computation of distances. To address this issue, we propose a lightweight adaptive random testing (Lw‐ARTDL) method for DL systems. In our improved algorithm, we employ the K‐Means technique to divide the entire test suite into several subsets. Then, for a candidate test case, we only calculate distances between it and the test cases within the category to which it belongs. This partition strategy ensures that the selected test cases are more representative while significantly reducing the computational cost. To validate the proposed algorithm, the comparison experiments between Lw‐ARTDL and the original ARTDL algorithm are conducted on two typical DL systems. The experimental results show that Lw‐ARTDL significantly reduces the overhead of failure detection, and exhibits stronger failure detection capability compared to ARTDL in most similarity metrics.

show abstract

Section: Related Workmentioning

confidence: 99%

A lightweight adaptive random testing method for deep learning systems

Mao,

Song,

Chen

2023

Softw Pract Exp

View full text Add to dashboard Cite

show abstract

“…Another research [16] discusses the use of clustering by grouping the resulting failures into smaller clusters, whereas successful results group into larger clusters. Instead of examining all the test outputs, the oracle problem may be reduced significantly by only examining the outputs of the smaller clusters.…”

Section: Related Workmentioning

confidence: 99%

“…In addition, there are researches in automating the test oracle problem, in [1], they discuss the automation methods they used, which include N-Version Diverse Systems, and M-Model Program Testing, Decision In our research, we discussed an alternative approach to alleviating the current test oracle problem, more specifically, we identified potential output anomalies. Similar to [16] and [17], our goal is to separate failures and successes into different clusters. However, in our approach, multiple clusters are created, each with its own distinct behavior.…”

Section: Related Workmentioning

confidence: 99%

A Clustering-Based Approach to Identifying Potential Output Anomalies with Combinatorial Testing for Web Applications

Wei,

Lee

2023

Preprint

View full text Add to dashboard Cite

The process of testing software is examining the artifacts and behavior of the software under test by validating and verifying it. Testing can provide valuable information about the software. Given an input to the system, there is a challenge of distinguishing correct from potentially incorrect behavior, which is the test oracle problem. Although the combinatorial testing approach can effectively reduce the number of test cases, the manual inspection of the outputs is still required. In this paper, we propose an approach to automatically classify the return pages with conducting combinatorial testing based on page similarities and hierarchical clustering, by which potential output anomalies can be identified. In the experiment, we apply our approach to a real-world e-commerce website, an anomaly can be found with the visualization of the clustering hierarchy, which suggests the potential of the applicability of the approach.

show abstract

“…Almaghairbe and his colleagues [43] proposed test oracles by clustering failures. They utilized anomaly detection techniques using software's input/output pairs.…”

Section: Introductionmentioning

confidence: 99%

Constructing Automated Test Oracle for Low Observable Software

et al. 2019

View full text Add to dashboard Cite

The application of machine learning techniques for constructing automated test oracles has been successful in recent years. However, existing machine learning based oracles are characterized by a number of de ciencies when applied to software systems with low observability, such as embedded software, cyber-physical systems, multimedia software programs, and computer games. This paper proposes a new black box approach to construct automated oracles that can be applied to software systems with low observability. The proposed approach employs an Arti cial Neural Network algorithm that uses input values and corresponding pass/fail outcomes of the program under test as the training set. To evaluate the performance of the proposed approach, extensive experiments were carried out on several benchmarks. The results manifest the applicability of the proposed approach to software systems with low observability and its higher accuracy than a well-known machine learning based method. This study also assessed the e ect of di erent parameters on the accuracy of the proposed approach.

show abstract

Building Test Oracles by Clustering Failures

Cited by 3 publications

References 19 publications

A lightweight adaptive random testing method for deep learning systems

A lightweight adaptive random testing method for deep learning systems

A Clustering-Based Approach to Identifying Potential Output Anomalies with Combinatorial Testing for Web Applications

Constructing Automated Test Oracle for Low Observable Software

Contact Info

Product

Resources

About