Domain-Specific Fixes for Flaky Tests with Wrong Assumptions on Underdetermined Specifications

Zhang, Peilun; Jiang, Yanjie; Wei, Anjiang; Stodden, Victoria; Marinov, Darko; Shi, August

doi:10.1109/icse43902.2021.00018

Cited by 23 publications

(6 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Fazzini et al [12] developed a tool to automatically generate test mocks [52] for mobile applications. Zhang et al [57] proposed a technique for repairing flaky tests related to non-deterministic specifications. Web E2e Flaky Tests.…”

Section: Discussionmentioning

confidence: 99%

WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

Liu,

Song,

Fang

et al. 2024

Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

Web end-to-end (e2e) testing evaluates the workflow of a web application. It simulates real-world user scenarios to ensure the application flows behave as expected. However, web e2e tests are notorious for being flaky, i.e., the tests can produce inconsistent results despite no changes to the code. One common type of flakiness is caused by nondeterministic execution orders between the test code and the client-side code under test. In particular, UI-based flakiness emerges as a notably prevalent and challenging issue to fix because the test code has limited knowledge about the client-side code execution. In this paper, we propose WEFix, a technique that can automatically generate fix code for UI-based flakiness in web e2e testing. The core of our approach is to leverage browser UI changes to predict the client-side code execution and generate proper wait oracles. We evaluate the effectiveness and efficiency of WEFix against 122 web e2e flaky tests from seven popular real-world projects. Our results show that WEFix dramatically reduces the overhead (from 3.7× to 1.25×) while achieving a high correctness (98%). CCS CONCEPTS• Software and its engineering → Software verification and validation.

show abstract

Section: Discussionmentioning

confidence: 99%

WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

Liu,

Song,

Fang

et al. 2024

Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

show abstract

“…This category accounts for seven diferent root causes that may lead to non-determinism in the build execution used for CPS development. Flakiness related to non-determinism during test execution [89] has been largely studied [16,46,48,58,90] and approaches to detect and cope with it have been proposed [47,49,59,65,87]. While similar to conventional software, dependency installation within the pipeline (PC 22 ) may result in pipelines having a laky behavior, e.g., for O 4 łROS uses GitHub repositories for dependency resolution so when GitHub or the repositories are down our build jobs will fail due to the impossibility of resolving dependenciesž, or else little control over external resources (PC 26 ), e.g., łthe most important root cause we experienced is related to the load on the server-sidež, the root causes behind laky behavior in CPSs may be diferent from conventional software.…”

Section: Challengementioning

confidence: 99%

Continuous Integration and Delivery Practices for Cyber-Physical Systems: An Interview-Based Study

Zampetti

Tamburri

Panichella

et al. 2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Continuous Integration and Delivery (CI/CD) practices have shown several benefits for software development and operations, e.g., faster release cycles and early discovery of defects. For Cyber-Physical System (CPS) development, CI/CD can help achieving required goals, such as high dependability, yet it may be challenging to apply. This paper empirically investigates challenges, barriers, and their mitigation occurring when applying CI/CD practices to develop CPSs in 10 organizations working in 8 different domains. The study has been conducted through semi-structured interviews, by applying an open card sorting procedure together with a member-checking survey within the same organizations, and by validating the results through a further survey involving 55 professional developers. The study reveals several peculiarities in the application of CI/CD to CPSs. These include the need for (i) combining continuous and periodic builds, while balancing the use of Hardware-in-the-Loop (HiL) and simulators; (ii) coping with difficulties in software deployment (iii) accounting for simulators and HiL differing in their behavior; and (vi) combining hardware/software expertise in the development team. Our findings open the road towards recommenders aimed at supporting the setting and evolution of CI/CD pipelines, as well as university curricula requiring interdisciplinarity, such as knowledge about hardware, software, and their interplay.

show abstract

“…It searches for tests from the same test suite that can make the order-dependent tests pass and reuses their code to generate the patch. DexFix [40] targets at unordered collections (e.g., HashMap and HashSet) and adopts a number of simple heuristics to fix these cases. FLEX [41] resolves the problems caused by the algorithmic randomness in ML algorithms.…”

Section: A Flaky Test Studiesmentioning

confidence: 99%

Peeler: Learning to Effectively Predict Flakiness without Running Tests

Qin

Wang

Liu

et al. 2022

2022 IEEE International Conference on Software Maintenance and Evolution (ICSME)

View full text Add to dashboard Cite

Regression testing is a widely adopted approach to expose change-induced bugs as well as to verify the correctness/robustness of code in modern software development settings. Unfortunately, the occurrence of flaky tests leads to a significant increase in the cost of regression testing and eventually reduces the productivity of developers (i.e., their ability to find and fix real problems). State-of-the-art approaches leverage dynamic test information obtained through expensive re-execution of test cases to effectively identify flaky tests. Towards accounting for scalability constraints, some recent approaches have built on static test case features, but fall short on effectiveness. In this paper, we introduce PEELER, a new fully static approach for predicting flaky tests through exploring a representation of test cases based on the data dependency relations. The predictor is then trained as a neural network based model, which achieves at the same time scalability (because it does not require any test execution), effectiveness (because it exploits relevant test dependency features), and practicality (because it can be applied in the wild to find new flaky tests). Experimental validation on 17,532 test cases from 21 Java projects shows that PEELER outperforms the state-of-the-art FlakeFlagger by around 20 percentage points: we catch 22% more flaky tests while yielding 51% less false positives. Finally, in a live study with projects in-the-wild, we reported to developers 21 flakiness cases, among which 12 have already been confirmed by developers as being indeed flaky.

show abstract

Domain-Specific Fixes for Flaky Tests with Wrong Assumptions on Underdetermined Specifications

Cited by 23 publications

References 28 publications

WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

Continuous Integration and Delivery Practices for Cyber-Physical Systems: An Interview-Based Study

Peeler: Learning to Effectively Predict Flakiness without Running Tests

Contact Info

Product

Resources

About