Surveying the Developer Experience of Flaky Tests

Parry, Owain; Kapfhammer, Gregory M.; Hilton, Michael; McMinn, Phil

doi:10.1109/icse-seip55303.2022.9793965

Cited by 3 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Luo et al [45] conducted the first empirical study on flaky tests. They found asynchronous waiting and concurrency to be the most common root causes-which other studies confirmed [29], [46], [47], [56], [57]-and recommended that detection techniques should focus on these. We take this observation into account by considering the difference between the tests' mean passing and failing durations and find that flaky failures tend to take longer than failures caused by regressions, which we suspect to be the result of waiting for asynchronous processes.…”

Section: Related Workmentioning

confidence: 87%

“…The second feature is the difference between the mean duration of passing and failing runs (mean duration diff ). Previous studies have shown that both concurrency and asynchronous waiting are major causes of flakiness [29], [45]- [47]. Since these aspects have significant impact on a test's duration through scheduling and timeouts, we suspect that they create characteristic patterns, by which we might be able to detect them.…”

Section: B Test Durationmentioning

confidence: 91%

See 1 more Smart Citation

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Gruber¹,

Heine²,

Oster³

et al. 2023

Preprint

View full text Add to dashboard Cite

Non-deterministically behaving test cases cause developers to lose trust in their regression test suites and to eventually ignore failures. Detecting flaky tests is therefore a crucial task in maintaining code quality, as it builds the necessary foundation for any form of systematic response to flakiness, such as test quarantining or automated debugging. Previous research has proposed various methods to detect flakiness, but when trying to deploy these in an industrial context, their reliance on instrumentation, test reruns, or language-specific artifacts was inhibitive. In this paper, we therefore investigate the prediction of flaky tests without such requirements on the underlying programming language, CI, build or test execution framework. Instead, we rely only on the most commonly available artifacts, namely the tests' outcomes and durations, as well as basic information about the code evolution to build predictive models capable of detecting flakiness. Furthermore, our approach does not require additional reruns, since it gathers this data from existing test executions. We trained several established classifiers on the suggested features and evaluated their performance on a large-scale industrial software system, from which we collected a data set of 100 flaky and 100 non-flaky test-and code-histories. The best model was able to achieve an F1-score of 95.5 % using only 3 features: the tests' flip rates, the number of changes to source files in the last 54 days, as well as the number of changed files in the most recent pull request.

show abstract

Section: Related Workmentioning

confidence: 87%

Section: B Test Durationmentioning

confidence: 91%

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Gruber¹,

Heine²,

Oster³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Using the same taxonomy defined by Luo [24], Eck et al [25] classified 200 flaky tests and identified four new causes of flakiness. Over the years, several surveys were carried on to identify the sources, impacts and existing strategies to mitigate flakiness by interrogating developers and practitioners [6]- [8], [26]. Parry et al presented the state of the art of academic research in another survey [27].…”

Section: Related Workmentioning

confidence: 99%

FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning

Amal

Haben

Habchi

et al. 2023

2023 IEEE/ACM International Conference on Automation of Software Test (AST)

View full text Add to dashboard Cite

Flaky tests are tests that yield different outcomes when run on the same version of a program. This nondeterministic behaviour plagues continuous integration with false signals, wasting developers' time and reducing their trust in test suites. Studies highlighted the importance of keeping tests flakiness-free. Recently, the research community has been pushing towards the detection of flaky tests by suggesting many static and dynamic approaches. While promising, those approaches mainly focus on classifying tests as flaky or not and, even when high performances are reported, it remains challenging to understand the cause of flakiness. This part is crucial for researchers and developers that aim to fix it. To help with the comprehension of a given flaky test, we propose FlakyCat, the first approach to classify flaky tests based on their root cause category. FlakyCat relies on CodeBERT for code representation and leverages Siamese networks to train a multi-class classifier. We train and evaluate FlakyCat on a set of 451 flaky tests collected from open-source Java projects. Our evaluation shows that FlakyCat categorises flaky tests accurately, with an F1 score of 73%. Furthermore, we investigate the performance of our approach for each category, revealing that Async waits, Unordered collections and Time-related flaky tests are accurately classified, while Concurrency-related flaky tests are more challenging to predict. Finally, to facilitate the comprehension of FlakyCat's predictions, we present a new technique for CodeBERT-based model interpretability that highlights code statements influencing the categorization.

show abstract

Exploring Tools for Flaky Test Detection, Correction, and Mitigation: A Systematic Mapping Study

Martins,

Alves,

Lima

et al. 2024

Anais Do IX Simpósio Brasileiro De Testes De Software Sistemático E Automatizado (SAST 2024)

View full text Add to dashboard Cite

Flaky tests, characterized by their non-deterministic behavior, present significant challenges in software testing. These tests exhibit uncertain results, even when executed on unchanged code. In the context of industrial projects that widely adopt continuous integration, the impact of flaky tests becomes critical. With thousands of tests, a single flaky test can disrupt the entire build and release process, leading to delays in software deliveries. In our study, we conducted a systematic mapping to investigate tools related to flaky tests. From a pool of 37 research papers, we identified 30 tools specifically designed for detecting, mitigating, and repairing flakiness in automated tests. Our analysis provides an overview of these tools, highlighting their objectives, techniques, and approaches. Additionally, we delve into the highest-level characteristics of these tools, including the causes they address. Notably, approximately 46% of the tools focus on tackling test order dependency issues, while a substantial majority (70%) of the tools are analyzed in the context of the Java programming language. These findings serve as valuable insights for two key groups of stakeholders: (Software Testing Community:) Researchers and practitioners can leverage this knowledge to enhance their understanding of flaky tests and explore effective mitigation strategies; (Tool Developers:) The compilation of available tools offers a centralized resource for selecting appropriate solutions based on specific needs. By addressing flakiness, we aim to improve the reliability of automated testing, streamline development processes, and foster confidence in software quality.

show abstract

Surveying the Developer Experience of Flaky Tests

Cited by 3 publications

References 34 publications

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

Practical Flaky Test Prediction using Common Code Evolution and Test History Data

FlakyCat: Predicting Flaky Tests Categories using Few-Shot Learning

Exploring Tools for Flaky Test Detection, Correction, and Mitigation: A Systematic Mapping Study

Contact Info

Product

Resources

About