Simulating the Effect of Test Flakiness on Fault Localization Effectiveness

Vancsics, Béla; Gergely, Tamás; Beszedes, Arpad

doi:10.1109/vst50071.2020.9051636

Cited by 9 publications

(8 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sometimes, a test case may pose an issue called ''test flakiness'', which refers to a test case with a non-deterministic result. In other words, sometimes it passes and sometimes it fails on the same code depending on unknown circumstances [84]. This issue negatively impacts the effectiveness of SBFL techniques as it provides misleading signals during the fault localization process [85].…”

Section: Test Flakinessmentioning

confidence: 99%

See 1 more Smart Citation

A Survey of Challenges in Spectrum-Based Software Fault Localization

Sarhan

Beszedes

2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

In software debugging, fault localization is the most difficult, expensive, tedious, and timeconsuming task, particularly for large-scale software systems. This is due to the fact that it requires significant human participation and it is difficult to automate its sub-tasks. Therefore, there is a high demand for automatic fault localization techniques that can help software engineers effectively find the locations of faults with minimal human intervention. This has led to the proposal of implementing different types of such techniques. However, Spectrum Based Fault Localization (SBFL) is considered amongst the most prominent techniques in this respect due to its efficiency and effectiveness. In SBFL, the probability of each program element (e.g., statement, block, or function) being faulty is calculated based on the results of executing test cases and their corresponding code coverage information. However, SBFL techniques are not yet widely adopted in the industry. The rationale behind this is that they pose a number of issues and their performance is affected by several influential factors. For example, the characteristics of bugs, target programs, test suites, and supporting tools make their effectiveness differ dramatically from one case to another. There are massive studies on SBFL that cover its usage, formulas, performance, etc. So far, no dedicated survey points out comprehensively the issues of SBFL. In this paper, various SBFL challenges and issues have been identified, categorized, and discussed alongside many directions. Also, the paper raises awareness of the works being achieved to address the identified issues and suggests some potential solutions too.INDEX TERMS Program spectra, spectrum based fault localization, software testing, challenges and issues, survey.

show abstract

Section: Test Flakinessmentioning

confidence: 99%

“…In [96], several formulas generated by genetic algorithms have been evaluated, and it has been found that the GP13 formula is one of the best performing formulas of its kind. In [84], it has been found that some SBFL spectra formulas (e.g., Tarantula) are more sensitive to the issue of test flakiness than others.…”

Section: N Spectra Formulas Selectionmentioning

confidence: 99%

A Survey of Challenges in Spectrum-Based Software Fault Localization

Sarhan

Beszedes

2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the projects we analysed, there are 70 commits in total that fits into the async wait category, such as the following example from In this test, the IPC message was sent before it was fully loaded, so the history did not contain everything it should have, and cannot clear all the pages, leading to non-deterministic outcomes. 6 In OS X, events for fs.watch() might only start showing up after a delay. To work around that, there is a timer in the test that delays the writing of the file by 100ms, but in some cases this might not be enough.…”

Section: A Rq1: What Are the Main Causes Of Flaky Tests In Javascript...mentioning

confidence: 99%

“…This can have a negative impact on development, product quality, and delivery [1]- [3]. Flakiness also impacts testing-related activities such as test suite minimization [4] and test parallelization [5], as well as techniques that rely on testing such as fault localization [6] or program repair [7].…”

Section: Introductionmentioning

confidence: 99%

An Empirical Study of Flaky Tests in JavaScript

Hashemi

Tahir

Rasheed

2022

View full text Add to dashboard Cite

Flaky tests (tests with non-deterministic outcomes) can be problematic for testing efficiency and software reliability. Flaky tests in test suites can also significantly delay software releases. There have been several studies that attempt to quantify the impact of test flakiness in different programming languages (e.g., Java and Python) and application domains (e.g., mobile and GUI-based). In this paper, we conduct an empirical study of the state of flaky tests in JavaScript. We investigate two aspects of flaky tests in JavaScript projects: the main causes of flaky tests in these projects and common fixing strategies. By analysing 452 commits from large, top-scoring JavaScript projects from GitHub, we found that flakiness caused by concurrency-related issues (e.g., async wait, race conditions or deadlocks) is the most dominant reason for test flakiness. The other top causes of flaky tests are operating system-specific (e.g., features that work on specific OS or OS versions) and network stability (e.g., internet availability or bad socket management). In terms of how flaky tests are dealt with, the majority of those flaky tests (>80%) are fixed to eliminate flaky behaviour and developers sometimes skip, quarantine or remove flaky tests.

show abstract

“…Test flakiness may defeat the purpose of the regression test via slowing down software deployment. Flaky tests may adversely affect different test processes such as fault localization 2 and automated fix detection 3…”

Section: Introductionmentioning

confidence: 99%

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

et al. 2020

View full text Add to dashboard Cite

A flaky test is a test that may lead to different results in different runs on a single code under test without any change in the test code. Test flakiness is a noxious phenomenon that slows down software deployment, and increases the expenditures in a broad spectrum of platforms such as software‐defined networks and Internet of Things environments. Industrial institutes and labs have conducted a whole lot of research projects aiming at tackling this problem. Although this issue has been receiving more attention from academia in recent years, the academic research community is still behind the industry in this area. A systematic review and trend analysis on the existing approaches for detecting and root causing flaky tests can pave the way for future research on this topic. This can help academia keep pace with industrial advancements and even lead the research in this field. This article first presents a comprehensive review of recent achievements of the industry as well as academia regarding the detection and mitigation of flaky tests. In the next step, recent trends in this line of research are analyzed and a roadmap is established for future research.

show abstract

Simulating the Effect of Test Flakiness on Fault Localization Effectiveness

Cited by 9 publications

References 20 publications

A Survey of Challenges in Spectrum-Based Software Fault Localization

A Survey of Challenges in Spectrum-Based Software Fault Localization

An Empirical Study of Flaky Tests in JavaScript

Root causing, detecting, and fixing flaky tests: State of the art and future roadmap

Contact Info

Product

Resources

About