The effects of change decomposition on code review - a controlled experiment

Abstract

Background. Code review is a cognitively demanding and time-consuming process. Previous qualitative studies hinted at how decomposing change sets into multiple yet internally coherent ones would improve the reviewing process. So far, no quantitative analysis of this hypothesis has been provided. Aims. (1) Quantitatively measure the effects of change decomposition on the outcome of code review (in terms of number of found defects, wrongly reported issues, suggested improvements, time, and understanding); (2) Qualitatively analyze how subjects approach the review and navigate the code, building knowledge and addressing existing issues, in large vs. decomposed changes. Method. Controlled experiment using the pull-based development model involving 28 software developers among professionals and graduate students. Results. Change decomposition leads to fewer wrongly reported issues, influences how subjects approach and conduct the review activity (by increasing context-seeking), yet impacts neither understanding the change rationale nor the number of found defects. Conclusions. Change decomposition reduces the noise for subsequent data analyses but also significantly supports the tasks of the developers in charge of reviewing the changes. As such, commits belonging to different concepts should be separated, adopting this as a best practice in software engineering.

Information Needs in Contemporary Code Review

“…For instance, Herzig and Zeller [30] proposed an automated approach relying on static and dynamic analysis to identify which code changes should be separated; Yamauchi suggested a clustering algorithm tuned to identify unrelated changes in a commit message [69]; and Dias et al [22] proposed a methodology to untangle code changes at a finer-granularity, i.e., by selecting the single statement of a code review that should be placed in other commits. More recently researchers also proposed untangling techniques tailored explicitly to code review [5,58] and conducted the first experiments to measure the effects of tangled code changes on code review [21,58] substantiating the value of separating unrelated changes. Despite these advances in splitting algorithms and their immediate practical value, no commercial code review tool offers this feature.…”

The effects of change decomposition on code review—a controlled experiment

“…To regain some degree of control, participants ran all the tasks in a provided virtual machine available in our replication package (di Biase et al, 2018). Moreover, we recorded the screencast of the experiment, therefore not leaving space to misaligned results and mitigating issues of incorrect interpretation.…”

“…The participants were asked to conduct the following four tasks. Further details are available in the online appendix (di Biase et al, 2018).…”

“…These changes can be inspected in the online appendix (di Biase et al, 2018) and have been chosen to meet the same criteria used by Herzig, Just & Zeller (2016) when choosing tangled changes. Changes proposed can be classified as refactoring and enhancement.…”

“…The design of an asynchronous controlled experiment to assess the benefits of change decomposition in code review using pull requests, available for replication (di Biase et al, 2018); Empirical evidence that change decomposition in the pull-based review environment leads to fewer false positives.…”