A Study of Regression Test Selection in Continuous Integration Environments

Yu, Tingting; Wang, Ting

doi:10.1109/issre.2018.00024

Cited by 11 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the codebase can be extremely large, and it might not be possible to execute the entire test suite. Subsequently, regression test selection approaches have been investigated to make continuous integration testing more costeffective [34,49,82,83,91,102].…”

Section: Regression Test Selection In Continuous Integration Environm...mentioning

confidence: 99%

“…We selected this second context as an extension to our conference-version paper [6] due to several reasons. Firstly, it is a context that has been widely investigated during the last few years in the context of regression testing [15,34,49,54,61,82,83,91,102]. This is, to a large extent, due to the importance that CI environments pose in current software engineering companies to develop their software systems.…”

Section: Regression Test Selection In Continuous Integration Environm...mentioning

confidence: 99%

See 1 more Smart Citation

Some Seeds Are Strong: Seeding Strategies for Search-based Test Case Selection

Arrieta

Valle

Agirre

et al. 2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

The time it takes software systems to be tested is usually long. Search-based test selection has been a widely investigated technique to optimize the testing process. In this paper, we propose a set of seeding strategies for the test case selection problem that generate the initial population of pareto-based multi-objective algorithms, with the goals of (1) helping to find an overall better set of solutions and (2) enhancing the convergence of the algorithms. The seeding strategies were integrated with four state-of-the-art multi-objective search algorithms and applied into two contexts where regression-testing is paramount: (1) Simulation-based testing of Cyber-Physical Systems and (2) Continuous Integration. For the first context, we evaluated our approach by using six fitness function combinations and six independent case studies, whereas in the second context we derived a total of six fitness function combinations and employed four case studies. Our evaluation suggests that some of the proposed seeding strategies are indeed helpful for solving the multi-objective test case selection problem. Specifically, the proposed seeding strategies provided a higher convergence of the algorithms towards optimal solutions in 96% of the studied scenarios and an overall cost-effectiveness with a standard search budget in 85% of the studied scenarios.

show abstract

Section: Regression Test Selection In Continuous Integration Environm...mentioning

confidence: 99%

Section: Regression Test Selection In Continuous Integration Environm...mentioning

confidence: 99%

Some Seeds Are Strong: Seeding Strategies for Search-based Test Case Selection

Arrieta

Valle

Agirre

et al. 2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

show abstract

“…We follow prior RTS research by analyzing tests at class (or file) rather than module or method granularity level[27,70,78].…”

mentioning

confidence: 99%

Empirically evaluating readily available information for regression test optimization in continuous integration

Elsner¹,

Hauer²,

Pretschner³

et al. 2021

Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis

View full text Add to dashboard Cite

Regression test selection (RTS) and prioritization (RTP) techniques aim to reduce testing efforts and developer feedback time after a change to the code base. Using various information sources, including test traces, build dependencies, version control data, and test histories, they have been shown to be effective. However, not all of these sources are guaranteed to be available and accessible for arbitrary continuous integration (CI) environments. In contrast, metadata from version control systems (VCSs) and CI systems are readily available and inexpensive. Yet, corresponding RTP and RTS techniques are scattered across research and often only evaluated on synthetic faults or in a specific industrial context. It is cumbersome for practitioners to identify insights that apply to their context, let alone to calibrate associated parameters for maximum cost-effectiveness. This paper consolidates existing work on RTP and unsafe RTS into an actionable methodology to build and evaluate such approaches that exclusively rely on CI and VCS metadata. To investigate how these approaches from prior research compare in heterogeneous settings, we apply the methodology in a large-scale empirical study on a set of 23 projects covering 37,000 CI logs and 76,000 VCS commits. We find that these approaches significantly outperform established RTP baselines and, while still triggering 90% of the failures, we show that practitioners can expect to save on average 84% of test execution time for unsafe RTS. We also find that it can be beneficial to limit training data, features from test history work better than change-based features, and, somewhat surprisingly, simple and well-known heuristics often outperform complex machine-learned models. CCS CONCEPTS• Software and its engineering → Software testing and debugging.

show abstract

“…Yu et al [77] evaluated method-level and class-level static RTS in continuous integration environments. Class-level RTS was determined to be more practical and time-saving than method-level RTS.…”

Section: Related Workmentioning

confidence: 99%

“…Several other RTS approaches [27,45,81] were recently proposed to make RTS more cost-effective for modern software systems. FLiRTS 2 follows this recent trend [27,31,44,45,77] and focuses on class-level RTS.…”

Section: Related Workmentioning

confidence: 99%

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

et al. 2021

View full text Add to dashboard Cite

Regression test selection (RTS) approaches reduce the cost of regression testing of evolving software systems. Existing RTS approaches based on UML models use behavioral diagrams or a combination of structural and behavioral diagrams. However, in practice, behavioral diagrams are incomplete or not used. In previous work, we proposed a fuzzy logic based RTS approach called FLiRTS that uses UML sequence and activity diagrams. In this work, we introduce FLiRTS 2, which drops the need for behavioral diagrams and relies on system models that only use UML class diagrams, which are the most widely used UML diagrams in practice. FLiRTS 2 addresses the unavailability of behavioral diagrams by classifying test cases using fuzzy logic after analyzing the information commonly provided in class diagrams. We evaluated FLiRTS 2 on UML class diagrams extracted from 3331 revisions of 13 open-source software systems, and compared the results with those of code-based dynamic (Ekstazi) and static (STARTS) RTS approaches. The average test suite reduction using FLiRTS 2 was 82.06%. The average safety violations of FLiRTS 2 with respect to Ekstazi and STARTS were 18.88% and 16.53%, respectively. FLiRTS 2 selected on average about 82% of the test cases that were selected by Ekstazi and STARTS. The average precision violations of FLiRTS 2 with respect to Ekstazi and STARTS were 13.27% and 9.01%, respectively. The average mutation score of the full test suites was 18.90%; the standard deviation of the reduced test suites from the average deviation of the mutation score for each subject was 1.78% for FLiRTS 2, 1.11% for Ekstazi, and 1.43% for STARTS. Our experiment demonstrated that the performance of FLiRTS 2 is close to the state-of-art tools for code-based RTS but requires less information and performs the selection in less time.

show abstract

A Study of Regression Test Selection in Continuous Integration Environments

Cited by 11 publications

References 20 publications

Some Seeds Are Strong: Seeding Strategies for Search-based Test Case Selection

Some Seeds Are Strong: Seeding Strategies for Search-based Test Case Selection

Empirically evaluating readily available information for regression test optimization in continuous integration

Bridging the model-to-code abstraction gap with fuzzy logic in model-based regression test selection

Contact Info

Product

Resources

About