Flaky test detection in Android via event order exploration

Dong, Zhao; Tiwari, Abhishek; Yu, Xiao; Roychoudhury, Abhik

doi:10.1145/3468264.3468584

Cited by 18 publications

(5 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There have also been a number of tools that have been targeted to detect certain types of flaky tests such as DeFlaker [28], RootFinder [29], iFixFlakies [30], SHAKER [31] and FlakeScanner [32].…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

An Empirical Study of Flaky Tests in JavaScript

Hashemi

Tahir

Rasheed

2022

View full text Add to dashboard Cite

Flaky tests (tests with non-deterministic outcomes) can be problematic for testing efficiency and software reliability. Flaky tests in test suites can also significantly delay software releases. There have been several studies that attempt to quantify the impact of test flakiness in different programming languages (e.g., Java and Python) and application domains (e.g., mobile and GUI-based). In this paper, we conduct an empirical study of the state of flaky tests in JavaScript. We investigate two aspects of flaky tests in JavaScript projects: the main causes of flaky tests in these projects and common fixing strategies. By analysing 452 commits from large, top-scoring JavaScript projects from GitHub, we found that flakiness caused by concurrency-related issues (e.g., async wait, race conditions or deadlocks) is the most dominant reason for test flakiness. The other top causes of flaky tests are operating system-specific (e.g., features that work on specific OS or OS versions) and network stability (e.g., internet availability or bad socket management). In terms of how flaky tests are dealt with, the majority of those flaky tests (>80%) are fixed to eliminate flaky behaviour and developers sometimes skip, quarantine or remove flaky tests.

show abstract

Section: Related Workmentioning

confidence: 99%

“…The developer noted the following in a pull request 32 : "...there are a couple of tests that are consistently flaky on arm (both Linux and Windows variants). This PR disables those flakes so that we can get our ARM CI to the point where it can be relied on for PR validation instead of maintainers ignoring it."…”

Section: Rq1 Findingsmentioning

confidence: 99%

An Empirical Study of Flaky Tests in JavaScript

Hashemi

Tahir

Rasheed

2022

View full text Add to dashboard Cite

show abstract

“…Several tools target concurrency as a source of flakiness. FlakeShovel [36] targets event races that can cause test flakiness by exploring different yet feasible event execution orders, however this is limited to GUI tests in Android apps. Shaker [37] exposes flakiness by adding noise to the environment in the form of tasks that also stress the CPU and memory whilst the test suite is executed.…”

Section: A Flaky Test Detection Techniquesmentioning

confidence: 99%

Flaky Test Sanitisation via On-the-Fly Assumption Inference for Tests with Network Dependencies

Dietrich¹,

Rasheed²,

Tahir³

2022

Preprint

View full text Add to dashboard Cite

Flaky tests cause significant problems as they can interrupt automated build processes that rely on all tests succeeding and undermine the trustworthiness of tests. Numerous causes of test flakiness have been identified, and program analyses exist to detect such tests. Typically, these methods produce advice to developers on how to refactor tests in order to make test outcomes deterministic.We argue that one source of flakiness is the lack of assumptions that precisely describe under which circumstances a test is meaningful. We devise a sanitisation technique that can isolate flaky tests quickly by inferring such assumptions on-the-fly, allowing automated builds to proceed as flaky tests are ignored. We demonstrate this approach for Java and Groovy programs by implementing it as extensions for three popular testing frameworks (JUnit4, JUnit5 and Spock) that can transparently inject the inferred assumptions. If JUnit5 is used, those extensions can be deployed without refactoring project source code.We demonstrate and evaluate the utility of our approach using a set of six popular real-world programs, addressing known test flakiness issues in these programs caused by dependencies of tests on network availability. We find that our method effectively sanitises failures induced by network connectivity problems with high precision and recall.Index Terms-flaky tests, non-determinism, regression testing3 This refers to the number of .class files found containing methods annotated as tests, methods with @Test or @ParameterizedTest annotations 4 Standard here refers to the assumptions and assertions defined as static methods in org.junit.jupiter.api.Assumptions (JU-nit5), org.junit.Assume (JUnit4), org.junit.jupiter.api.-Assertions (JUnit5) and org.junit.Assert (JUnit4) respectively, we are counting the call sites for those methods in the respective classes.

show abstract

“…Till now, several frameworks have been proposed to detect flaky tests by performing test reruns in various environments [12], [13], [14], [15], [16], [17]. Other dynamic approaches have been proposed where flaky tests are detected by monitoring test coverage [18], instrumenting programs [4] or varying execution orders [19], [20]. Because such dynamic approaches are expensive in terms of time and computational cost, static approaches are also being considered: Pinto et al [21] extracted code vocabularies from test code and used machine learning algorithms to predict flaky tests based on the assumption that flaky test code follows certain grammatical patterns.…”

Section: Introductionmentioning

confidence: 99%

Peeler: Learning to Effectively Predict Flakiness without Running Tests

Qin

Wang

Liu

et al. 2022

2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

Regression testing is a widely adopted approach to expose change-induced bugs as well as to verify the correctness/robustness of code in modern software development settings. Unfortunately, the occurrence of flaky tests leads to a significant increase in the cost of regression testing and eventually reduces the productivity of developers (i.e., their ability to find and fix real problems). State-of-the-art approaches leverage dynamic test information obtained through expensive re-execution of test cases to effectively identify flaky tests. Towards accounting for scalability constraints, some recent approaches have built on static test case features, but fall short on effectiveness. In this paper, we introduce PEELER, a new fully static approach for predicting flaky tests through exploring a representation of test cases based on the data dependency relations. The predictor is then trained as a neural network based model, which achieves at the same time scalability (because it does not require any test execution), effectiveness (because it exploits relevant test dependency features), and practicality (because it can be applied in the wild to find new flaky tests). Experimental validation on 17,532 test cases from 21 Java projects shows that PEELER outperforms the state-of-the-art FlakeFlagger by around 20 percentage points: we catch 22% more flaky tests while yielding 51% less false positives. Finally, in a live study with projects in-the-wild, we reported to developers 21 flakiness cases, among which 12 have already been confirmed by developers as being indeed flaky.

show abstract

Flaky test detection in Android via event order exploration

Cited by 18 publications

References 45 publications

An Empirical Study of Flaky Tests in JavaScript

An Empirical Study of Flaky Tests in JavaScript

Flaky Test Sanitisation via On-the-Fly Assumption Inference for Tests with Network Dependencies

Peeler: Learning to Effectively Predict Flakiness without Running Tests

Contact Info

Product

Resources

About