When testing meets code review

Abstract

Automated testing is considered an essential process for ensuring software quality. However, writing and maintaining high-quality test code is challenging and frequently considered of secondary importance. For production code, many open source and industrial software projects employ code review, a well-established software quality practice, but the question remains whether and how code review is also used for ensuring the quality of test code. The aim of this research is to answer this question and to increase our understanding of what developers think and do when it comes to reviewing test code. We conducted both quantitative and qualitative methods to analyze more than 300,000 code reviews, and interviewed 12 developers about how they review test files. This work resulted in an overview of current code reviewing practices, a set of identified obstacles limiting the review of test code, and a set of issues that developers would like to see improved in code review tools. The study reveals that reviewing test files is very different from reviewing production files, and that the navigation within the review itself is one of the main issues developers currently face. Based on our findings, we propose a series of recommendations and suggestions for the design of tools and future research. CCS CONCEPTS• Software and its engineering → Software testing and debugging;

Modern Code Reviews—Survey of Literature and Practice

“…However, test code is not discussed as much as production code [224]. When reviewing test code, developers face challenges such as lack of testing context, poor navigation support (between test and production code), unrealistic time constraints imposed by management, and poor knowledge of good reviewing and testing practices by novice developers [224]. Test-driven code review (TDR) is the practice of reviewing test code before production code and studied in a controlled experiment and survey [226].…”

“…Review scope -Another aspect of preparing for code review is to decide which artifacts to review, i.e., the review scope. Test code is seldom reviewed and is not considered worthy of review [224,226]. However, the survey respondents consider test code review one of the most important research topics (c.f., Figure 7c).…”

How Does This New Developer Test Fit In? A Visualization to Understand Amplified Test Cases

“…From related works, we know that developers who inspect a test case during code review are interested in the code under test [7]. They rely on source code to understand the system under test [16], [17] and should be supported while building up the mental context between test code and code under test [18].…”

“…During the study, we go through three example test cases with each participant. The examples are amplified test cases which were generated by the test amplification plugin TestCube 7 to improve the test suite of the project javapoet 8 . We select this project for our study as we expect our participants to be familiar with its domain: generating Java source files.…”

“…Developer tests-xUnit test programs which developers use to check the behavior of their code [1]-have become a cornerstone in assuring the quality of today's software systems [2]. As test suites are growing in number and size, understanding test cases one has not written themselves is becoming more and more important, for example, (a) when trying to understand a failing test [3], (b) when using developer tests as a form of executable documentation [4]- [6], (c) when test cases are submitted for code review [7], or (d) when determining whether to add an automatically generated test case to the test suite, e.g., checking whether the captured behavior is correct [8]- [10].…”