Scalable Approaches for Test Suite Reduction

Cruciani, Emilio; Miranda, Breno; Verdecchia, Roberto; Bertolino, Antonia

doi:10.1109/icse.2019.00055

Cited by 43 publications

(30 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For constructing sets of interesting test cases we will most likely need a multi-objective formulation that combines diversity of sets of values [9] and derivatives/quotients. Practical work on how to select interesting and relevant distance functions for particular purposes and how to speed up distance calculations are also important and recent advances show promise [6].…”

Section: Discussionmentioning

confidence: 99%

Towards Automated Boundary Value Testing with Program Derivatives and Search

Feldt

Dobslaw

2019

Search-Based Software Engineering

View full text Add to dashboard Cite

A natural and often used strategy when testing software is to use input values at boundaries, i.e. where behavior is expected to change the most, an approach often called boundary value testing or analysis (BVA). Even though this has been a key testing idea for long it has been hard to clearly define and formalize. Consequently, it has also been hard to automate. In this research note we propose one such formalization of BVA by, in a similar way as to how the derivative of a function is defined in mathematics, considering (software) program derivatives. Critical to our definition is the notion of distance between inputs and outputs which we can formalize and then quantify based on ideas from Information theory. However, for our (black-box) approach to be practical one must search for test inputs with specific properties. Coupling it with search-based software engineering is thus required and we discuss how program derivatives can be used as and within fitness functions. This brief note does not allow a deeper, empirical investigation but we use a simple illustrative example throughout to introduce the main ideas. By combining program derivatives with search, we thus propose a practical as well as theoretically interesting technique for automated boundary value (analysis and) testing.

show abstract

Section: Discussionmentioning

confidence: 99%

Towards Automated Boundary Value Testing with Program Derivatives and Search

Feldt

Dobslaw

2019

Search-Based Software Engineering

View full text Add to dashboard Cite

show abstract

“…In previous studies [19], [20] we have shown that test code similarity can provide an effective instrument for test suite prioritization and reduction. Inspired by such studies, in this work we leverage test code similarity for identifying flaky tests.…”

Section: Introductionmentioning

confidence: 94%

“…Similarly to what we did in a previous work [20], we model the tests in T as points in an n-dimensional vector space using the bag-of-words model [42]: each test case t is represented as the multiset (i.e., a set that allows multiple instances of its elements) of the lowercase tokens composing its source code, split by whitespace characters and punctuation. We purposely decided not to manipulate the input data, e.g., we did not exclude comments, as they are exclusively the original ones written by developers and not added by researchers a posteriori, reflecting the real-world nature of our experimental subjects.…”

Section: A: Vector Space Modelingmentioning

confidence: 99%

Know You Neighbor: Fast Static Prediction of Test Flakiness

et al. 2021

Self Cite

View full text Add to dashboard Cite

Context: Flaky tests plague regression testing in Continuous Integration environments by slowing down change releases and wasting testing time and effort. Despite the growing interest in mitigating the burden of test flakiness, how to efficiently and effectively detect flaky tests is still an open problem.Objective: In this study, we present and evaluate FLAST, an approach designed to statically predict test flakiness. FLAST leverages vector-space modeling, similarity search, dimensionality reduction, and k-Nearest Neighbor classification in order to timely and efficiently detect test flakiness. Method: In order to gain insights into the efficiency and effectiveness of FLAST, we conduct an empirical evaluation of the approach by considering 13 real-world projects, for a total of 1,383 flaky and 26,702 nonflaky tests. We carry out a quantitative comparison of FLAST with the state-of-the-art methods to detect test flakiness, by considering a balanced dataset comprising 1,402 real-world flaky and as many non-flaky tests. Results: From the results we observe that the effectiveness of FLAST is comparable with the state-of-theart, while providing considerable gains in terms of efficiency. In addition, the results demonstrate how by tuning the threshold of the approach FLAST can be made more conservative, so to reduce false positives, at the cost of missing more potentially flaky tests. Conclusion:The collected results demonstrate that FLAST provides a fast, low-cost and reliable approach that can be used to guide test rerunning, or to gate the inclusion of new potentially flaky tests.

show abstract

“…For tracing code coverage, we have used Cobertura, 3 which gives us the ids of the executed code lines for each test case run. Producing coverage information may introduce up to 30% of time overhead during test execution (Cruciani et al 2019). Therefore, the coverage information is produced and collected only for the classes of the system under test, i.e.…”

Section: A Comprehensive Approach For the Automatic Generation And Rementioning

confidence: 99%

“…In Cruciani et al (2019), the authors reduce test suites by computing the similarity among test cases and take the most distant tests as these should be the ones that potentially show faults on several parts of the system. Their reduction approach need not executing tests on the system, hence it performs quickly.…”

Section: Test Suite Reduction Selection and Prioritisationmentioning

confidence: 99%

REDUNET: reducing test suites by integrating set cover and network-based optimization

2020

View full text Add to dashboard Cite

The availability of effective test suites is critical for the development and maintenance of reliable software systems. To increase test effectiveness, software developers tend to employ larger and larger test suites. The recent availability of software tools for automatic test generation makes building large test suites affordable, therefore contributing to accelerating this trend. However, large test suites, though more effective, are resources and time consuming and therefore cannot be executed frequently. Reducing them without decreasing code coverage is a needed compromise between efficiency and effectiveness of the test, hence enabling a more regular check of the software under development. We propose a novel approach, namely REDUNET, to reduce a test suite while keeping the same code coverage. We integrate this approach in a complete framework for the automatic generation of efficient and effective test suites, which includes test suite generation, code coverage analysis, and test suite reduction. Our approach formulates the test suite reduction as a set cover problem and applies integer linear programming and a network-based optimisation, which takes advantage of the properties of the control flow graph. We find the optimal set of test cases that keeps the same code coverage in fractions of seconds on real software projects and test suites generated automatically by Randoop. The results on ten real software systems show that the proposed approach finds the optimal minimisation and achieves up to 90% reduction and more than 50% reduction on all systems under analysis. On the largest project our reduction algorithm performs more than three times faster than both integer linear programming alone and the state-of-the-art heuristic Harrold Gupta Soffa.

show abstract

Scalable Approaches for Test Suite Reduction

Cited by 43 publications

References 34 publications

Towards Automated Boundary Value Testing with Program Derivatives and Search

Towards Automated Boundary Value Testing with Program Derivatives and Search

Know You Neighbor: Fast Static Prediction of Test Flakiness

REDUNET: reducing test suites by integrating set cover and network-based optimization

Contact Info

Product

Resources

About