Negative Effects of Bytecode Instrumentation on Java Source Code Coverage

Tengeri, Dávid; Horváth, Ferenc; Beszedes, Arpad; Gergely, Tamás; Gyimóthy, Tibor

doi:10.1109/saner.2016.61

Cited by 13 publications

(15 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Contrary to what would be expected, this granularity level also involves diculties in the interpretation of code coverage, which was the main motivation for our research. In particular, we found signicant dierences between dierent code coverage measurement tools for Java congured for method level analysis (Tengeri et al, 2016).…”

Section: Dierent Types and Levels Of Code Coveragementioning

confidence: 91%

“…However, in most cases the application of code coverage is on the source code, hence it is worthwhile to investigate and compare the two approaches. In earlier work (Tengeri et al, 2016), we investigated these two types of code coverage measurement approaches via two representative tools on a set of open source Java programs. We found that there were many deviations in the raw coverage results due to the various technical and conceptual dierences of the instrumentation methods.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Code coverage differences of Java bytecode and source code instrumentation tools

et al. 2017

View full text Add to dashboard Cite

Many software testing elds, like white-box testing, test case generation, test prioritization and fault localization, depend on code coverage measurement. If used as an overall completeness measure, the minor inaccuracies of coverage data reported by a tool do not matter that much; however, in certain situations they can lead to serious confusion. For example, a code element that is falsely reported as covered can introduce false condence in the test. This work investigates code coverage measurement issues for the Java programming language. For Java, the prevalent approach to code coverage measurement is using bytecode instrumentation due to its various benets over source code instrumentation. As we have experienced, bytecode instrumentation-based code coverage tools produce dierent results than source code instrumentation-based ones in terms of the reported items as covered. We report on an empirical study to compare the code coverage results provided by tools using the dierent instrumentation types for Java coverage measurement on the method level. In particular, we want to nd out how much a bytecode instrumentation approach is inaccurate compared to a source code instrumentation method. The dierences are systematically investigated both in quantitative (how much the outputs dier) and in qualitative terms (what the causes for the dierences are). In addition, the impact on test prioritization and test suite reduction a possible application of coverage measurement is investigated in more detail as well. Keywords Code coverage • white-box testing • Java bytecode instrumentation • source code instrumentation • coverage tools • empirical study The nal publication is available at Springer via

show abstract

Section: Dierent Types and Levels Of Code Coveragementioning

confidence: 91%

Section: Introductionmentioning

confidence: 99%

Code coverage differences of Java bytecode and source code instrumentation tools

et al. 2017

View full text Add to dashboard Cite

show abstract

“…We modified the build processes of the systems to produce method level coverage information using the Clover coverage measurement tool. 2 This tool is based on source-code instrumentation and gives more precise information about source code entities than tools based on bytecode instrumentation Tengeri et al 2016).…”

Section: Subject Programs and Detection Frameworkmentioning

confidence: 99%

Differences between a static and a dynamic test-to-code traceability recovery method

et al. 2018

View full text Add to dashboard Cite

Recovering test-to-code traceability links may be required in virtually every phase of development. This task might seem simple for unit tests thanks to two fundamental unit testing guidelines: isolation (unit tests should exercise only a single unit) and separation (they should be placed next to this unit). However, practice shows that recovery may be challenging because the guidelines typically cannot be fully followed. Furthermore, previous works have already demonstrated that fully automatic test-to-code traceability recovery for unit tests is virtually impossible in a general case. In this work, we propose a semi-automatic method for this task, which is based on computing traceability links using static and dynamic approaches, comparing their results and presenting the discrepancies to the user, who will determine the final traceability links based on the differences and contextual information. We define a set of discrepancy patterns, which can help the user in this task. Additional outcomes of analyzing the discrepancies are structural unit testing issues and related refactoring suggestions. For the static test-to-code traceability, we rely on the physical code structure, while for the dynamic, we use code coverage information. In both cases, we compute combined test and code clusters which represent sets of mutually traceable elements. We also present an empirical study of the method involving 8 non-trivial open source Java systems.

show abstract

“…Finally, Tengeri et al (2016) argue that Jacoco which is based on bytecode instrumentation may produce erroneous results compared to source code instrumentations methods. Jacoco misses some really covered methods.…”

Section: Internal Validitymentioning

confidence: 99%

Test case selection in industry: an analysis of issues related to static approaches

et al. 2016

View full text Add to dashboard Cite

Automatic testing constitutes an important part of everyday development practice. Worldline, a major IT company, is creating more and more tests to ensure the good behaviour of its applications and gain in efficiency and quality. But running all these tests may take hours. This is especially true for large systems involving, for example, the deployment of a web server or the communication with a database. For this reason tests are not launched as often as they should and are mostly run at night. The company wishes to improve its development and testing process by giving to developers rapid feedback after a change. An interesting solution is to reduce the number of tests to run by identifying only those exercising the piece of code changed. Two main approaches are proposed in the literature: static and dynamic. The static approach creates a model of the source code and explores it to find links between changed methods and tests. The dynamic approach records invocations of methods during the execution of test scenarios. Before deploying a test case selection solution, Worldline created a partnership with us to investigate the situation in its projects and to evaluate these approaches on three industrial, closed source, cases to understand the strengths and weaknesses of each solution. We propose a classification of problems that may arise when trying to identify the tests that cover a method. We give concrete examples of these problems and list some possible solutions. We also evaluate other issues such as the impact on the results of the frequency of modification of methods or considering groups of methods instead of single ones. We found that solutions must be combined to obtain better results, and, problems have different impacts on projects. Considering commits instead of individual methods tends to worsen the results, perhaps due to their large size.

show abstract

Negative Effects of Bytecode Instrumentation on Java Source Code Coverage

Cited by 13 publications

References 23 publications

Code coverage differences of Java bytecode and source code instrumentation tools

Code coverage differences of Java bytecode and source code instrumentation tools

Differences between a static and a dynamic test-to-code traceability recovery method

Test case selection in industry: an analysis of issues related to static approaches

Contact Info

Product

Resources

About