Prioritizing manual test cases in rapid release environments

Hemmati, Hadi; Fang, Zhihan; Mäntylä, Martti; Adams, Bram

doi:10.1002/stvr.1609

Cited by 28 publications

(33 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hemmati et al [18] investigated the effectiveness of three black-box TCP techniques on Firefox before and after the transition to rapid releases. The authors concluded that 5 HBTP is far more effective than other comparable techniques in rapid releases, although the same conclusion does not hold in traditional development environments.…”

Section: Related Work 221 Test Prioritization In CI Environmentsmentioning

confidence: 99%

“…Therefore, the tests that failed in previous releases have a much higher probability of failing again in the current release. This is perhaps due to the recency of historical knowledge, which explains its effectiveness in a rapid-release environment, rather than other changes in the development process [18]. There is also a number of studies that are particularly focused on HBTP.…”

Section: Related Work 221 Test Prioritization In CI Environmentsmentioning

confidence: 99%

“…The underlying hypothesis is that similar test cases will likely exercise the same part of the system and detect the same fault; thus, a diverse set of test cases must be performed to detect more faults [21]. This hypothesis has been further investigated by other researchers, e.g., [33,34,35,36,18,20,37,38]. The implication for TCP is that higher priority must be assigned to those test cases that are most different from those already prioritized.…”

Section: Diversity-based Test Prioritizationmentioning

confidence: 99%

“…The implication for TCP is that higher priority must be assigned to those test cases that are most different from those already prioritized. Diversity-based TCP can be implemented using different methods and on different levels, e.g., source-code behind test cases [20], method calls [36], topic models extracted from test cases [38], or English texts of manual test cases [18].…”

Section: Diversity-based Test Prioritizationmentioning

confidence: 99%

“…Even though diversity-based TCP has been proposed and used previously, it has not been widely studied in combination with HBTP and in CI environments. Hemmati et al [18] previously investigated a history-based diversity approach in Firefox project. They used previous failure knowledge in combination with a single similarity metric, Manhattan distance, using the English texts of manual test cases.…”

Section: Research Gapmentioning

confidence: 99%

See 4 more Smart Citations

Test prioritization in continuous integration environments

Haghighatkhah

Mäntylä

Oivo

et al. 2018

Journal of Systems and Software

View full text Add to dashboard Cite

Two heuristics namely diversity-based (DBTP) and history-based test prioritization (HBTP) have been separately proposed in the literature. Yet, their combination has not been widely studied in continuous integration (CI) environments. The objective of this study is to catch regression faults earlier, allowing developers to integrate and verify their changes more frequently and continuously. To achieve this, we investigated six open-source projects, each of which included several builds over a large time period. Findings indicate that previous failure knowledge seems to have strong predictive power in CI environments and can be used to effectively prioritize tests. HBTP does not necessarily need to have large data, and its effectiveness improves to a certain degree with larger history interval. DBTP can be used effectively during the early stages, when no historical data is available, and also combined with HBTP to improve its effectiveness. Among the investigated techniques, we found that history-based diversity using NCD Multiset is superior in terms of effectiveness but comes with relatively higher overhead in terms of method execution time. Test prioritization in CI environments can be effectively performed with negligible investment using previous failure knowledge, and its effectiveness can be further improved by considering dissimilarities among the tests. arXiv:1809.00143v1 [cs.SE] 1 Sep 2018 functionality. RT is widely used in practice; it is common to have a dedicated regression test suite that is often run in its entirety [4].A test suite for enterprise-sized applications often includes thousands of test cases, the execution of which requires several hours or even days. For instance, the JOnAS Java EE middleware comprises 2,689 test cases [5]. Applying them all to its 16 configurations results in running 43,024 test cases. Furthermore, the cost of RT increases over the time with the increase in system size. Memon et. al [6] observed linear growth in both code submission rate and size of regression test suite, so incurring significant expenses to keep the RT running. The software engineering literature has proposed many techniques to improve RT processes. Test suite minimization (TSM) [7] aims to eliminate test cases from a test suite with a specific objective, i.e., removing obsolete or redundant test cases. Several experiments have been reported in the literature with differing conclusions regarding the impact of TSM on the fault detection capability of a test suite, e.g., [8,9,10]. However, the common understanding is that TSM may compromise such capability. Test case prioritization (TCP) [11], on the other hand, is concerned with the ideal ordering of test cases to maximize desirable properties (i.e., early fault detection). From the perspective of fault detection, TCP seems to be a safe approach because it does not eliminate test cases and simply permutes them within the test suite.The intersection of CI and RT poses great challenges for the software development industry. The testing budget is often...

show abstract

Section: Related Work 221 Test Prioritization In CI Environmentsmentioning

confidence: 99%

Section: Related Work 221 Test Prioritization In CI Environmentsmentioning

confidence: 99%

Section: Diversity-based Test Prioritizationmentioning

confidence: 99%

Section: Diversity-based Test Prioritizationmentioning

confidence: 99%

Section: Research Gapmentioning

confidence: 99%

See 3 more Smart Citations

Test prioritization in continuous integration environments

Haghighatkhah

Mäntylä

Oivo

et al. 2018

Journal of Systems and Software

View full text Add to dashboard Cite

show abstract

Test Case Prioritization Using Test Similarities

Haghighatkhah

Mäntylä

Oivo

et al. 2018

Product-Focused Software Process Improvement

View full text Add to dashboard Cite

A classical heuristic in software testing is to reward diversity, which implies that a higher priority must be assigned to test cases that differ the most from those already prioritized. This approach is commonly known as similarity-based test prioritization (SBTP) and can be realized using a variety of techniques. The objective of our study is to investigate whether SBTP is more effective at finding defects than random permutation, as well as determine which SBTP implementations lead to better results. To achieve our objective, we implemented five different techniques from the literature and conducted an experiment using the defects4j dataset, which contains 395 real faults from six real-world open-source Java programs. Findings indicate that running the most dissimilar test cases early in the process is largely more effective than random permutation (Vargha-Delaney A [VDA]: 0.76-0.99 observed using normalized compression distance). No technique was found to be superior with respect to the effectiveness. Locality-sensitive hashing was, to a small extent, less effective than other SBTP techniques (VDA: 0.38 observed in comparison to normalized compression distance), but its speed largely outperformed the other techniques (i.e., it was approximately 5-111 times faster). Our results bring to mind the well-known adage, "don't put all your eggs in one basket". To effectively consume a limited testing budget, one should spread it evenly across different parts of the system by running the most dissimilar test cases early in the testing process.

show abstract

Automating system test case classification and prioritization for use case-driven testing in product lines

et al. 2020

View full text Add to dashboard Cite

Product Line Engineering (PLE) is a crucial practice in many software development environments where software systems are complex and developed for multiple customers with varying needs. At the same time, many development processes are use case-driven and this strongly influences their requirements engineering and system testing practices. In this paper, we propose, apply, and assess an automated system test case classification and prioritization approach specifically targeting system testing in the context of use case-driven development of product families. Our approach provides: (i) automated support to classify, for a new product in a product family, relevant and valid system test cases associated with previous products, and (ii) automated prioritization of system test cases using multiple risk factors such as fault-proneness of requirements and requirements volatility in a product family. Our evaluation was performed in the context of an industrial product family in the automotive domain. Results provide empirical evidence that we propose a practical and beneficial way to classify and prioritize system test cases for industrial product lines.

show abstract

Prioritizing manual test cases in rapid release environments

Cited by 28 publications

References 37 publications

Test prioritization in continuous integration environments

Test prioritization in continuous integration environments

Test Case Prioritization Using Test Similarities

Automating system test case classification and prioritization for use case-driven testing in product lines

Contact Info

Product

Resources

About