No abstract

Many software testing elds, like white-box testing, test case generation, test prioritization and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false condence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benets over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce dierent results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the dierent instrumentation types for Java coverage measurement on the method level. In particular, we want to nd out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The dierences are systematically investigated both in quantitative (how much the outputs dier) and in qualitative terms (what the causes for the dierences are). In addition, the impact on test prioritization and test suite reduction a possible application of coverage measurement is investigated in more detail as well. Keywords Code coverage • white-box testing • Java bytecode instrumentation • source code instrumentation • coverage tools • empirical study The nal publication is available at Springer via

Code coverage measurement is an important element in white-box testing, both in industrial practice and academic research. Other related areas are highly dependent on code coverage as well, including test case generation, test prioritization, fault localization, and others. Inaccuracies of a code coverage tool sometimes do not matter that much but in certain situations they can lead to serious confusion. For Java, the prevalent approach to code coverage measurement is to use bytecode instrumentation due to its various benefits over source code instrumentation. However, if the results are to be mapped back to source code this may lead to inaccuracies due to the differences between the two program representations. In this paper, we systematically investigate the amount of differences in the results of these two Java code coverage approaches, enumerate the possible reasons and discuss the implications on various applications. For this purpose, we relied on two widely used tools to represent the two approaches and a set of benchmark programs from the open source domain.

No abstract

Code coverage is successfully used to guide white box test design and evaluate the respective test completeness. However, simple overall coverage ratios are often not precise enough to effectively help when a (regression) test suite needs to be reassessed and evolved after software change. We present an approach for test suite assessment and improvement that utilizes code coverage information, but on a more detailed level and adds further evaluation aspects derived from the coverage. The main use of the method is to aid various test suite evolution situations such as removal, refactoring and extension of test cases as a result of code change or test suite efficiency enhancement. We define various metrics to express different properties of test suites beyond simple code coverage ratios, and present the assessment and improvement process as an iterative application of different improvement goals and more specific sub-activities. The method is demonstrated by applying it to improve the tests of one of our experimental systems.