A learning algorithm for optimizing continuous integration development and testing practice

Abstract

Summary Continuous integration, at its core, includes a set of practices that aim to prevent and reduce the cost of software integration issues by merging working software copies often. Regression testing is considered a good practice in software development with continuous integration, which ensures that code changes are not negatively affecting software functionality. As, nowadays, software development is carried out iteratively, with small code increments continuously developed and regression tested, it is of critical importance that continuous regression testing is time efficient. However, in practice, regression testing is often long lasting and faces scalability problems as software grows larger or as software changes are made more frequently. One contributing factor to these issues is test redundancy, which causes the same software functionality being tested multiple times across a test suite. In large‐scale software, especially highly configurable software, redundancy in continuous regression testing can significantly grow the size of test suites and negatively affect the cost effectiveness of continuous integration. This paper presents a practical learning algorithm for optimizing continuous integration testing by reducing ineffective test redundancy in regression suites. The novelty of the algorithm lies in learning and predicting the fault‐detection effectiveness of continuous integration tests using historical test records and combining this information with coverage‐based redundancy metrics. The goal is to identify ineffective redundancy, which is maximally reduced in the resulting regression test suite, thus reducing test time and improving the performance of continuous integration. We apply and evaluate the algorithm in two industrial projects of continuous integration. The results show that the proposed algorithm can improve the efficiency of continuous integration practice in terms of decreasing test execution time by 38% on average compared to the industry practice of our case study and by 40% on average compared to the retest‐all approach. The results further demonstrate no significant reduction in fault‐detection effectiveness of continuous regression testing. This suggests that the proposed algorithm contributes to the state of the practice in the continuous integration development and testing of highly configurable systems.

Learning-based prioritization of test cases in continuous integration of highly-configurable software

“…Regarding TCP in CI environments we refer a recent mapping [30] that highlights some works for the HCS context [19,20,21,22]. The approach of Marijan et al [20,21] uses historical test data to determine an optimal order of test cases to ensure feature coverage, early fault detection and execution time.…”

“…The idea is to obtain a high fault detection rate and low test execution. Another work [19] uses the coverage matrix of test cases and the fault detection history to identify redundant test cases that are not likely to detect faults. Their method minimizes a test suite by excluding redundant test cases.…”

“…A recent mapping reports approaches that adapt TCP techniques for CI environments [30], some of them addressing the HCS context [19,21,22]. But most existing approaches present some limitations.…”

Efficient Regression Testing of Software Product Lines by Reducing Redundant Test Executions

“…Several other approaches [41][42][43][44] have been proposed to minimize the size of test suite by removing obsolete or redundant test cases. Baller et al [41] use an integer linear programming to minimize a test suite for a product family.…”

“…Their method optimizes a test suite using both a test minimization technique and a test prioritization technique so as to achieve a high fault detection rate and a low test execution time. Another work of Marijan et al [42] uses the coverage matrix of test cases and the fault detection history to identify redundant test cases that are not likely to detect faults. Their method minimizes a test suite by excluding redundant test cases from a test suite.…”