A program slicing-based method for effective detection of coincidentally correct test cases

Feyzi, Farid; Parsa, Saeed

doi:10.1007/s00607-018-0591-z

Cited by 26 publications

(10 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Feyzi [49], the author compares the proposed SBFL technique to six other techniques; however, when assessing the impact of CC, only Ochiai and CGT-FL are used. • The study by Feyzi [49] relied on three automated techniques [50][51][52][53][54][55][56] to identify the CC tests without differentiating between weak CC and strong CC, whereas in our study, the classification of the test cases as weak CC and strong CC was done by manually injecting code checkers [15]. This suggests that our classification is likely to be more accurate; furthermore, it differentiates between weak CC and strong CC.…”

Section: Related Workmentioning

confidence: 99%

How detrimental is coincidental correctness to coverage‐based fault detection and localization? An empirical study

Assi

Masri

Trad

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

According to the reachability-infection-propagation (RIP) model, three conditions must be satisfied for program failure to occur: (1) the defect's location must be reached, (2) the program's state must become infected and (3) the infection must propagate to the output. Weak coincidental correctness (or weak CC) occurs when the program produces the correct output, while condition (1) is satisfied but conditions (2) and (3) are not satisfied. Strong coincidental correctness (or strong CC) occurs when the output is correct, while both conditions (1) and ( 2) are satisfied but not (3). The prevalence of CC was previously recognized. In addition, the potential for its negative effect on spectrum-based fault localization (SBFL) was analytically demonstrated; however, this was not empirically validated. Using Defects4J, this paper empirically studies the impact of weak and strong CC on three well-researched coverage-based fault detection and localization techniques, namely, test suite reduction (TSR), test case prioritization (TCP) and SBFL. Our study, which involved 52 SBFL metrics, provides the following empirical evidence. (i) The negative impact of CC tests on TSR and TCP is very significant. In addition, cleansing the CC tests was observed to yield (a) a 100% TSR defect detection rate for all subject programs and (b) an improvement of TCP for over 92% of the subjects. (ii) The impact of CC tests on SBFL varies widely w.r.t. the metric used. The negative impact was strong for 11 metrics, mild for 37, non-measurable for 1 and non-existent for 3 metrics. Interestingly, the negative impact was mild for the 9 most popular and/or most effective SBFL metrics. In addition, cleansing the CC tests resulted in the deterioration of SBFL for a considerable number of subject programs. (iii) Increasing the proportion of CC tests has a limited impact on TSR, TCP and SBFL. Interestingly, for TSR and TCP and 11 SBFL metrics, small and large proportions of CC tests are strongly harmful. (iv) Lastly, weak and strong CC are equally detrimental in the context of TSR, TCP and SBFL.

show abstract

Section: Related Workmentioning

confidence: 99%

How detrimental is coincidental correctness to coverage‐based fault detection and localization? An empirical study

Assi

Masri

Trad

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

show abstract

“…In practice, software developers are unaware of how many faults caused the failures and usually target one fault at a time in their debugging [59]- [61]. That is, after the first fault has been successfully located and fixed, developers re-run test cases in order to detect the subsequent failures and then locate and fix the next fault.…”

Section: Case Studies a Benchmark Description And Experimental Setupmentioning

confidence: 99%

“…They proposed two techniques for cleansing test suites from coincidental correctness to improve SBFL. Following the work in [71], Feyzi, and Parsa [59] further classified test cases as failing or passing and presented a program slicing-based technique that identifies the subset of passing tests that are likely to be coincidentally correct in order to improve SBFL.…”

Section: Related Work a Studies On Improving Sbflmentioning

confidence: 99%

A Framework for Improving Fault Localization Effectiveness Based on Fuzzy Expert System

2021

View full text Add to dashboard Cite

Many spectrum-based fault localization (SBFL) techniques have been proposed in order to improve debugging efficiency. These SBFL techniques were designed according to different underlying assumptions and then adopt different fault locator functions to evaluate the likelihood of each statement being faulty, called suspiciousness. So far no single SBFL technique claims that it can outperform all of the others under every scenario. That is, the effectiveness of fault localization may vary considerably by just adopting a single SBFL technique. Due to the aforementioned reasons, this study presents a framework for improving fault localization effectiveness by using a Fuzzy Expert System (FES) to integrate different SBFL techniques. In the presented framework, the outputs of several SBFL techniques will be put into the fuzzification and then transferred to fuzzy input sets. After undergoing the fuzzy inference based on the given fuzzy rules, the fuzzy input sets will be transferred to a fuzzy output set. Finally, the fuzzy set will be transferred to a crisp output (called a weighted suspiciousness value). The code statements will then be ranked according to their weighted suspiciousness values. In other words, no additional instrumentations and analyses on the source code and the test suite are necessary for our approach. Our experiment results indicate that our FES-based framework is effective at combining the SBFL techniques from different equivalent groups and achieves high effectiveness on the nine subject programs. It is also noted that in the literature, most of the approaches that combine multiple SBFL techniques are learning-based and they are suitable for long-term projects with sufficient historical data. Since our approach does not reference historical data for model training, it can be applied to new software projects. Thus, the application scenarios of our approach should be complementary to those of the state-of-the-art learning-based approaches.INDEX TERMS Software debugging, fault localization, fuzzy expert system.

show abstract

“…Many experiments have been carried out using fault seeding to evaluate testing techniques in terms of their fault detection effectiveness [16,[38][39][40]. Mutation is performed using MILU [41] which mutates C programmes using various mutation operators which can be classified into four classes: statement, operator, variable and constant.…”

Section: Case Studymentioning

confidence: 99%

Bayes‐TDG: effective test data generation using Bayesian belief network: toward failure‐detection effectiveness and maximum coverage

Feyzi

Parsa

2018

IET softw.

Self Cite

View full text Add to dashboard Cite

This study presents a novel test data generation method called Bayes-TDG. It is based on principles of Bayesian networks and provides the possibility of making inference from probabilistic data in the model to increase the prime pathcoverage ratio for a given programme under test (PUT). In this regard, a new programme structure-based probabilistic network, TDG-NET, is proposed that is capable of modelling the conditional dependencies among the programme basic blocks (BBs) on one hand and conditional dependencies of the transitions between its BBs and input parameters on the other hand. To achieve failure-detection effectiveness, the authors propose a path selection strategy that works based on the predicted outcome of generated test cases. So, they mitigate the need for a human oracle, and the generated test suite could be directly used in fault localisation. Several experiments are conducted to evaluate the performance of Bayes-TDG. The results reveal that the method is promising and the generated test suite could be quite effective.

show abstract

A program slicing-based method for effective detection of coincidentally correct test cases

Cited by 26 publications

References 45 publications

How detrimental is coincidental correctness to coverage‐based fault detection and localization? An empirical study

How detrimental is coincidental correctness to coverage‐based fault detection and localization? An empirical study

A Framework for Improving Fault Localization Effectiveness Based on Fuzzy Expert System

Bayes‐TDG: effective test data generation using Bayesian belief network: toward failure‐detection effectiveness and maximum coverage

Contact Info

Product

Resources

About