Learning-based prioritization of test cases in continuous integration of highly-configurable software

Abstract

Continuous Integration (CI) is a practice widely adopted in the industry to allow frequent integration of code changes. During the CI process, many test cases are executed multiple times a day, subject to time constraints. In this scenario, a learning-based approach, named COLEMAN, has been successfully applied. COLEMAN allows earlier execution of the most promising test cases to reveal faults. This approach considers CI particularities such as time budget and volatility of test cases, related to the fact that test cases can be added/removed along the CI cycles. In the CI of Highly Configuration System (HCS), many product variants must be tested, each one with different configuration options, but having test cases that are common to or reused from other variants. In this context, we found, by analogy, another particularity, the volatility of variants, that is, some variants can be included/discontinued along CI cycles. Considering this context, this work introduces two strategies for the application of COLEMAN in the CI of HCS: the Variant Test Set Strategy (VTS) that relies on the test set specific for each variant, and the Whole Test Set Strategy (WST) that prioritizes the test set composed by the union of the test cases of all variants. Both strategies are evaluated in a real-world HCS, considering three test budgets. The results show that the proposed strategies are applicable regarding the time spent for prioritization. They perform similarly regarding early fault detection, but WTS better mitigates the problem of beginning without knowledge, and is more suitable when a new variant to be tested is added. CCS CONCEPTS • General and reference → Experimentation; • Software and its engineering → Software product lines; Software testing and debugging.

Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review

“…Table 12 summarizes the comparison baselines used to evaluate the ML-based TSP techniques proposed in the studies. We observe that the majority (nine) of the studies [7,10,21,24,25,26,31,32,35] compared their proposed techniques with a random approach, in which test cases are ordered randomly, or a cost-only approach, in which tests are ordered based on execution time [22]. In addition, there are three studies [15,20,28] in which the evaluation of the proposed techniques was compared to approaches that rank test cases using traditional quality metrics (e.g., fault history or coverage-based) and similarity metrics [29].…”

“…Three studies used a Multi-Armed Bandit (MAB) approach for TP [9,10,11]. Lima et al [9] used a MAB-based approach to address test case prioritization in a CI context (TCPCI).…”

“…Their results suggested that the MAB approach can fare similarly to the GA algorithm in 90% of the cases in terms of the percentage of faults detected, Root-Mean-Square-Error (RMSE), and Prioritization Time. In another work, Lima et al [10] also considered two strategies for TP using MAB in the CI of Highly Configurable Systems (HCS): the product Variant Test Set Strategy (VTS), which prioritizes the test set for each product variant in the HCS and the Whole Test Set Strategy (WTS), which prioritizes the test set based on all test cases for all product variants. While these two strategies fared similarly in terms of performance, WTS works well when testing new product variants, since it makes use of the historical information collected from the other ones.…”

“…Test execution times are also relevant since they can vary a great deal across test cases. [26, 28, 29] Open Source Repositories (e.g., GitHub & GitLab) [8,9,10,11,22,23] Infrastructure Supporting Experimentation [53] [27] Private/Local/Industrial Projects [15,17,18,21,24,25,30,34] -Average execution time of test cases per build. One of the main motivations of the TSP techniques is decreasing regression testing time.…”

“…However, we further excluded four of those studies as they report NAPFD values using line plots [7,12,13] or boxplots [27], making it difficult to estimate the average performance of the techniques. As a result, we considered ten studies [9,10,11,15,20,21,24,25,26,32] in which APFD/NAPFD values were explicitly and precisely reported. Given that the performance of an ML-based TSP technique can differ from one subject (i.e., software project) to another, we extracted individual APFD or NAPFD values for each subject in all primary studies.…”