A large-scale study of test coverage evolution

Abstract

Statement coverage is commonly used as a measure of test suite quality. Coverage is often used as a part of a code review process: if a patch decreases overall coverage, or is itself not covered, then the patch is scrutinized more closely. Traditional studies of how coverage changes with code evolution have examined the overall coverage of the entire program, and more recent work directly examines the coverage of patches (changed statements). We present an evaluation much larger than prior studies and moreover consider a new, important kind of change-coverage changes of unchanged statements. We present a large-scale evaluation of code coverage evolution over 7,816 builds of 47 projects written in popular languages including Java, Python, and Scala. We find that in large, mature projects, simply measuring the change to statement coverage does not capture the nuances of code evolution. Going beyond considering statement coverage as a simple ratio, we examine how the set of statements covered evolves between project revisions. We present and study new ways to assess the impact of a patch on a project's test suite quality that both separates coverage of the patch from coverage of the non-patch, and separates changes in coverage from changes in the set of statements covered. CCS CONCEPTS • Software and its engineering → Software testing and debugging;

Understanding flaky tests: the developer’s perspective

“…Memon and Cohen [26] described multiple negative effects that flaky tests can create, discovering that their occurrence may even lead to features not being included in a release as they were not able to be tested sufficiently beforehand [26]. Marinescu et al [25] and Hilton et al [20] analyzed the evolution of test suite coverage, reporting that the presence of flaky tests produces an intermittent variation of the branch coverage.…”

How bugs are born: a model to identify how bugs are introduced in software components

“…Moonen et al have shown that about 2/3 of the refactoring changes from Fowler et al (1999) can actually result in non-building test cases because the refactoring changes the original interface and the test code requires a change in the types of classes that were involved in the refactoring (Moonen et al 2008). In contrast, Hilton et al have recently performed a study on test coverage evolution using Continuous Integration builds (Beller et al 2017), reporting that this modern infrastructure eases building prior versions of a software project considerably (Hilton et al 2018). We could consider implementing these perfect tests by automatically generating them, e.g., using EvoSuite (Fraser and Arcuri 2013a;Palomba et al 2016).…”

Continuous Integration Theater

“…COVERALLS also provides an API in which it made available information about the branch, the total coverage, and the change in the coverage in a particular build. The data available on COVERALLS have also be used in other studies (e.g., [17]).…”

“…However, this is often non-trivial task (e.g., some projects fail, some projects require manual configuration, etc). Since recent related work is also employing COVERALLS (e.g., [17]), we opted to use the COVERALLS infrastructure to gather coverage information for this work as well.…”