Test case selection in industry: an analysis of issues related to static approaches

Abstract

Automatic testing constitutes an important part of everyday development practice. Worldline, a major IT company, is creating more and more tests to ensure the good behaviour of its applications and gain in efficiency and quality. But running all these tests may take hours. This is especially true for large systems involving, for example, the deployment of a web server or the communication with a database. For this reason tests are not launched as often as they should and are mostly run at night. The company wishes to improve its development and testing process by giving to developers rapid feedback after a change. An interesting solution is to reduce the number of tests to run by identifying only those exercising the piece of code changed. Two main approaches are proposed in the literature: static and dynamic. The static approach creates a model of the source code and explores it to find links between changed methods and tests. The dynamic approach records invocations of methods during the execution of test scenarios. Before deploying a test case selection solution, Worldline created a partnership with us to investigate the situation in its projects and to evaluate these approaches on three industrial, closed source, cases to understand the strengths and weaknesses of each solution. We propose a classification of problems that may arise when trying to identify the tests that cover a method. We give concrete examples of these problems and list some possible solutions. We also evaluate other issues such as the impact on the results of the frequency of modification of methods or considering groups of methods instead of single ones. We found that solutions must be combined to obtain better results, and, problems have different impacts on projects. Considering commits instead of individual methods tends to worsen the results, perhaps due to their large size.

A Class-level Test Selection Approach Toward Full Coverage For Continuous Integration

“…Note that we also check the changed class C i , which ensures to select the newly-added tests or changed tests. Following existing work [6], [20], if it contains at least one method (the method name started with "test" or the method has the annotation "@Test", "@BeforeClass", "@Before"), we regard it as a test class. If it is a test class, we will add it to the set of related tests T Set.…”

“…However, dynamic test dependencies for large projects may be time-consuming to collect, and for real-time systems, dynamic selection techniques may not be applicable, because code instrumentation for obtaining dependencies may cause timeouts or interrupt normal test run [5], [15]. Besides, for programs with nondeterminism (e.g., due to randomness or concurrency), dependencies collected dynamically may not cover all possible traces, leading to omission of necessary tests [5], [6]. Static selection technique does not require to execute tests; it uses static program analysis to infer the dependencies between changed code, affected code and test code [4], [5], [13], [18], [19].…”

“…It is easier to manipulate than dynamic technique thus draws more and more attention. However, the drawback of existing static approaches is that the selected test sets either cannot fully cover the necessary tests or are too redundant due to inefficient static analysis techniques [4]- [6], [13], [20].…”

Evaluating the efficiency of continuous testing during test-driven development

“…The presence of a type system 1 obviously helps to construct a reasonably accurate dependency graph. Nevertheless, because of language features like exceptions, type-casts, reflection, dynamic link libraries it is impossible to guarantee a complete call graph [16].…”

Usage of Tests in an Open-Source Community

“…This is, of course, a simplified example. In real cases, establishing dependencies from tests to code is made difficult by many adverse factors Blondeau et al [2016]. We now proceed to describe the previously published case study that we take inspiration from.…”