Know You Neighbor: Fast Static Prediction of Test Flakiness

Verdecchia, Roberto; Cruciani, Emilio; Miranda, Breno; Bertolino, Antonia

doi:10.1109/access.2021.3082424

Cited by 36 publications

(4 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[26]. This shows that the availability of language-specific artifacts is a concern not only to dynamic, but also to static and hybrid approaches, that use them to extract code smells [25], data dependencies [26], or the vocabulary of tests [27], [28], or to measure complexity metrics [24].…”

Section: A Using Instrumentation or Language-specific Artifacts To De...mentioning

confidence: 98%

“…To address this demand, researchers have proposed many flakiness detection approaches [15]- [28]. We found most of these difficult to implement in an industrial setting, since they rely on code instrumentation [15]- [20], which is not always implementable, demand multiple test reruns [19]- [23], which is computationally demanding, or require language-specific artifacts [24]- [28]. Our observation is underlined by a recent study that found the adoption rates of automated flakiness detection techniques among practitioners to be poor [29].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Gruber¹,

Heine²,

Oster³

et al. 2023

Preprint

View full text Add to dashboard Cite

Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test-and code-histories. The best model was able to achieve an F1-score of 95.5 % using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

show abstract

Section: A Using Instrumentation or Language-specific Artifacts To De...mentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Gruber¹,

Heine²,

Oster³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The flaky tests have gained wide attention in the CI field [37,38]. Many researchers have focused on the research about flaky tests, for example, flakiness predicting [39][40][41], the lifecycle of flaky tests [42] and the impact of flaky tests on TCP [1,10]. In our work, we compare the effectiveness of SatTCP and baseline approaches on both flaky tests and nonflaky tests, respectively, to investigate the impact of flaky tests on the performance of TCP methods.…”

Section: Tcp In CImentioning

confidence: 99%

Semantic‐aware two‐phase test case prioritization for continuous integration

Li,

Wang,

Wang

et al. 2023

Software Testing Verif & Rel

View full text Add to dashboard Cite

SummaryContinuous integration (CI) is a widely applied development practice to allow frequent integration of software changes, detecting early faults. However, extremely frequent builds consume amounts of time and resources in such a scenario. It is quite challenging for existing test case prioritization (TCP) to address this issue due to the time‐consuming information collection (e.g. test coverage) or inaccurately modelling code semantics to result in the unsatisfied prioritization. In this paper, we propose a semantic‐aware two‐phase TCP framework, named SatTCP, which combines the coarse‐grained filtering and fine‐grained prioritization to perform the precise TCP with low time costs for CI. It consists of three parts: (1) code representation, parsing the programme changes and test cases to obtain the code change and test case representations; (2) coarse‐grained filtering, conducting the preliminary ranking and filtering of test cases based on information retrieval; and (3) fine‐grained prioritization, training a pretrained Siamese language model based on the filtered test set to further sort the test cases via semantic similarity. We evaluate SatTCP on a large‐scale, real‐world dataset with cross‐project validation from fault detection efficiency and time costs and compare it with five baselines. The results show that SatTCP outperforms all baselines by 6.3%–45.6% for mean average percentage of fault detected per cost (APFDc), representing an obvious upward trend as the project scale increases. Meanwhile, SatTCP can reduce the real CI testing by 71.4%, outperforming the best baseline by 17.2% for time costs on average. Furthermore, we discuss the impact of different configurations, flaky tests and hybrid techniques on the performance of SatTCP, respectively.

show abstract

“…Silva et al[51] focus on detecting asynchronous wait and concurrency flakiness by introducing environment noise. Many works[1,9,21,39,55] apply machine learning for flakiness detection. Mitigation and Repair.…”

mentioning

confidence: 99%

WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

Liu,

Song,

Fang

et al. 2024

Proceedings of the ACM Web Conference 2024

View full text Add to dashboard Cite

Web end-to-end (e2e) testing evaluates the workflow of a web application. It simulates real-world user scenarios to ensure the application flows behave as expected. However, web e2e tests are notorious for being flaky, i.e., the tests can produce inconsistent results despite no changes to the code. One common type of flakiness is caused by nondeterministic execution orders between the test code and the client-side code under test. In particular, UI-based flakiness emerges as a notably prevalent and challenging issue to fix because the test code has limited knowledge about the client-side code execution. In this paper, we propose WEFix, a technique that can automatically generate fix code for UI-based flakiness in web e2e testing. The core of our approach is to leverage browser UI changes to predict the client-side code execution and generate proper wait oracles. We evaluate the effectiveness and efficiency of WEFix against 122 web e2e flaky tests from seven popular real-world projects. Our results show that WEFix dramatically reduces the overhead (from 3.7× to 1.25×) while achieving a high correctness (98%). CCS CONCEPTS• Software and its engineering → Software verification and validation.

show abstract

Know You Neighbor: Fast Static Prediction of Test Flakiness

Cited by 36 publications

References 46 publications

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Semantic‐aware two‐phase test case prioritization for continuous integration

WEFix: Intelligent Automatic Generation of Explicit Waits for Efficient Web End-to-End Flaky Tests

Contact Info

Product

Resources

About