Toolset and Program Repository for Code Coverage-Based Test Suite Analysis and Manipulation

2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

Horváth

Beszedes

et al. 2016

Self Cite

Code coverage measurement is an important element in white-box testing, both in industrial practice and academic research. Other related areas are highly dependent on code coverage as well, including test case generation, test prioritization, fault localization, and others. Inaccuracies of a code coverage tool sometimes do not matter that much but in certain situations they can lead to serious confusion. For Java, the prevalent approach to code coverage measurement is to use bytecode instrumentation due to its various benefits over source code instrumentation. However, if the results are to be mapped back to source code this may lead to inaccuracies due to the differences between the two program representations. In this paper, we systematically investigate the amount of differences in the results of these two Java code coverage approaches, enumerate the possible reasons and discuss the implications on various applications. For this purpose, we relied on two widely used tools to represent the two approaches and a set of benchmark programs from the open source domain.

Section: A Overviewmentioning

confidence: 99%

Negative Effects of Bytecode Instrumentation on Java Source Code Coverage

2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

Horváth

Beszedes

et al. 2016

Self Cite

“…Integration tests typically exercise code from multiple modules by definition, hence they are probably less relevant from this aspect. More details on the aforementioned metrics can be found elsewhere [6], [7], [8]. Table II shows our extended list of metrics obtained for the WebKit system, more precisely for each of the 9 functional units identified.…”

Section: Test Suite Assessment Metricsmentioning

confidence: 99%

“…Later, this approach was to measure the test suite of the WebKit project, a large industrially supported open source web browser layout engine [7]. For the analysis, we used the SoDA library [8].…”

Section: Introductionmentioning

confidence: 99%

Assessing the Test Suite of a Large System Based on Code Coverage, Efficiency and Uniqueness

Vidács

Horváth

2016 IEEE 23rd International Conference on Software Analysis, Evolution, and Reengineering (SANER)

et al. 2016

Self Cite

Regression test suites of evolving software systems play a key role in maintaining software quality throughout continuous changes. They need to be effective (in terms of detecting faults and helping their localization) and efficient (optimally sized and without redundancy) at the same time. However, test suite quality attributes are usually difficult to formalize and measure. In this paper, we rely on a recent approach for test suite assessment and improvement that utilizes code coverage information, but at a more detailed level, hence it adds further evaluation aspects derived from the coverage. The basic idea of the method is to decompose the test suite and the program code into coherent logical groups which are easier to analyze and understand. Several metrics are then computed from code coverage information to characterize the test suite and its constituents. We extend our previous study and employ derived coverage metrics (which express efficiency and uniqueness) to analyze the test suite of a large scale industrial open source system containing 27 000 test cases.

“…Next, we developed a supporting toolset (called SoDA) with optimized data structures and algorithms suitable to handle such industrial large and complex systems as WebKit [11], [12]. With the help of WebKit developers, we determined procedure level functional units (composed of functions and methods) along with the associated test groups and measured detailed code coverage information for the test suite.…”

Section: Motivation and Goalsmentioning

confidence: 99%

“…Our library and toolset called SoDA and TAM have been designed specifically to accomodate the proposed approach, and is freely available as open source [11], [12].…”

Section: Use Casesmentioning

confidence: 99%

Beyond code coverage — An approach for test suite assessment and improvement

2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW)

Beszedes

Gergely

et al. 2015

Self Cite

Code coverage is successfully used to guide white box test design and evaluate the respective test completeness. However, simple overall coverage ratios are often not precise enough to effectively help when a (regression) test suite needs to be reassessed and evolved after software change. We present an approach for test suite assessment and improvement that utilizes code coverage information, but on a more detailed level and adds further evaluation aspects derived from the coverage. The main use of the method is to aid various test suite evolution situations such as removal, refactoring and extension of test cases as a result of code change or test suite efficiency enhancement. We define various metrics to express different properties of test suites beyond simple code coverage ratios, and present the assessment and improvement process as an iterative application of different improvement goals and more specific sub-activities. The method is demonstrated by applying it to improve the tests of one of our experimental systems.