Towards a deeper understanding of test coverage

Abstract

Test coverage is traditionally considered as how much of the code is covered by the test suite in whole. However, test suites typically contain different types of tests with different roles, such as unit tests, integration tests and functional tests. As traditional measures of test coverage make no distinction between the different types of tests, the overall view of test coverage is limited to what is covered by the tests in general. This paper proposes a quantitative way to measure the test coverage of the different parts of the software at different testing levels. It is also shown how this information can be used in software maintenance and development to further evolve the test suite and the system under test. The technique is applied to an open-source project to show its application in practice.Traditional measures of test coverage focus on measuring how much of the total SUT has been exercised by the test suite. Various coverage measures include measures such as statement, path and decision coverage [1]. These traditional types of code coverage are useful for seeing which parts are not yet under test and for getting an overview of how much of the SUT is tested in general. However, once previously uncovered code is brought under test, traditional code coverage measures only tell us that the code is covered by some test in the test suite. These measures do not tell anything about the types of tests exercising the piece of code. Thus, if we want to understand better how the SUT is covered by different types of tests, the traditional view of test coverage does not provide a good view for this purpose.This paper proposes a way to get a deeper understanding of testing for the different parts of the SUT. Instead of considering how much of the SUT is covered by the test suite in total, it is considered how the different parts of the SUT are covered at the different levels of testing. A measure for the level of testing for the different parts of the SUT is defined and it is shown how this measure can be used during software maintenance and evolution to get a deeper understanding of the testing for the SUT and to evolve the existing test suite. The measure is applied to an open-source software (OSS) project to demonstrate its application in practice.This paper is structured as follows. The next section discusses the basic concepts and describes the measure used in this paper. Section 3 shows how the measure has been implemented in practice. Section 4 applies the measure to an OSS project and analyses the results. Section 5 discusses the benefits and problems in applying the measure. Section 6 reviews related work in literature. Finally, Section 7 discusses conclusions and future work.

Recovering traceability links between unit tests and classes under test: An improved method

“…They use the median number of executed methods per test as a good indicator if that test suite is well suited for documentation purposes or not. Kanstrén [25] associates testing levels with call trace information, to reveal the scope of test cases, identify shortcomings in the test suite or remove redundant test cases. In the following sections, we discuss the literature regarding the definition of methods for recovering traceability links between different types of artefacts.…”

Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining

“…As a consequence of these different testing strategies, the Test Coverage Evolution View cannot be used to compare across projects. Difference in testing strategy and the consequences on test coverage have also been described by Kanstrén, who proposes to measure test coverage separately for each level of testing (Kanstrén 2008).…”

Differences between a static and a dynamic test-to-code traceability recovery method

“…In this work, we concentrate on unit tests, in which case the traceability information is mostly encoded in the source code of the unit test cases, and usually no external documentation is available for this purpose. Traceability recovery for unit test may seem simple at first (Beck 2002;Demeyer et al 2002;Gaelli et al 2007); however, in reality, it is not (Gaelli et al 2005;Kanstrén 2008). Bruntink and Van Deursen (2004) illustrated the need and complexity of the test-to-code traceability.…”

“…For example, in the coding and testing phases, finding low-level code-related defects early is essential using appropriate unit tests, and in evolution, up to date unit tests are necessary for efficient regression testing. Several approaches have been proposed for this task (e.g., Qusef et al 2014;Gaelli et al 2005;Kanstrén 2008;Bruntink and Van Deursen 2004;Bouillon et al 2007). However, practice shows that the fully automatic recovery of traceability links is difficult, and the different approaches might produce different results (Rompaey and Demeyer 2009;Qusef et al 2010).…”