Analyzing regression test selection techniques

Abstract

Abstract-Regression testing is a necessary but expensive maintenance activity aimed at showing that code has not been adversely affected by changes. Regression test selection techniques reuse tests from an existing test suite to test a modified program. Many regression test selection techniques have been proposed; however, it is difficult to compare and evaluate these techniques because they have different goals. This paper outlines the issues relevant to regression test selection techniques, and uses these issues as the basis for a framework within which to evaluate the techniques. We illustrate the application of our framework by using it to evaluate existing regression test selection techniques. The evaluation reveals the strengths and weaknesses of existing techniques, and highlights some problems that future work in this area should address.

Improving Regression Testing Transparency and Efficiency with History-Based Prioritization -- An Industrial Case Study

“…There is however other views on what is a good selection (e.g. high inclusiveness or precision [22]) or prioritization (e.g. early code coverage) of test cases which we could not analyze in this study due to limitations of available data.…”

“…The metrics used for comparing the techniques are accepted by the research community and have been used in several previous studies for the same purposes [22], [7], [13]. There is however other views on what is a good selection (e.g.…”

Covrig: a framework for the analysis of code, test, and coverage evolution in real software

“…For example, the research community has invested a lot of effort in designing techniques for improving the testing of software patches, ranging from test suite prioritisation and selection algorithms [11,30,35] to program analysis techniques for test case generation and bug finding [1,2,20,21,27,28,36,40] to methods for surviving errors introduced by patches at runtime [14]. Many of these techniques depend on the existence of a manual test suite, sometimes requiring the availability of a test exercising the patch [24,37], sometimes making assumptions about the stability of program coverage or external behaviour over time [14,29], other times using it as a starting point for exploration [10,16,22,39], and often times employing it as a baseline for comparison [3,6,9,26].…”

High-Coverage Symbolic Patch Testing

“…This would be valuable for large patches, especially those touching multiple code areas. To this purpose, we envision leveraging test selection techniques [24]. Furthermore, we plan to improve our prototype by allowing it to automatically infer the patch location from a diff file.…”

“…[3,10,18,24]. These techniques are particularly useful for very large projects where running the entire test suite at each change of the system is infeasible (for example in the Windows operating system testing infrastructure [26]).…”