On the Optimal Order of Reading Source Code Changes for Review

Abstract

Change-based code review, e.g., in the form of pull requests, is the dominant style of code review in practice. An important option to improve review's efficiency is cognitive support for the reviewer. Nevertheless, review tools present the change parts under review sorted in alphabetical order of file path, thus leaving the effort of understanding the construction, connections, and logic of the changes on the reviewer. This leads to the question: How should a code review tool order the parts of a code change to best support the reviewer? We answer this question with a middle-range theory, which we generated inductively in a mixed methods study, based on interviews, an online survey, and existing findings from related areas. Our results indicate that an optimal order is mainly an optimal grouping of the change parts by relatedness. We present our findings as a collection of principles and formalize them as a partial order relation among review orders. Abstract-Change-based code review, e.g., in the form of pull requests, is the dominant style of code review in practice. An important option to improve review's efficiency is cognitive support for the reviewer. Nevertheless, review tools present the change parts under review sorted in alphabetical order of file path, thus leaving the effort of understanding the construction, connections, and logic of the changes on the reviewer. This leads to the question: How should a code review tool order the parts of a code change to best support the reviewer? We answer this question with a middle-range theory, which we generated inductively in a mixed methods study, based on interviews, an online survey, and existing findings from related areas. Our results indicate that an optimal order is mainly an optimal grouping of the change parts by relatedness. We present our findings as a collection of principles and formalize them as a partial order relation among review orders.

Associating working memory capacity and code change ordering with code review performance

“…For the effect of change part ordering, our findings are less conclusive: There is insufficient evidence to allow robust conclusions that change part ordering has an effect on code review performance. Our findings show that the general tendency and the qualitative data is compatible with the predictions in our previous work (Baum et al, 2017b), but we do not reach statistical significance. Not having attained the sample size indicated by power analysis could have had an effect on this result.…”

“…It helps checking for defects by avoiding speculative assumptions and by easing the spotting of inconsistencies." They present six principles for a good order (Baum et al, 2017b, pg. 5f):…”

“…A large portion of the research efforts on automated tools to help code reviewing is explicitly or implicitly based on the assumption that reducing the mental load of reviewers improves their code review performance (Baum et al, 2017b). Cognitive psychology uses the term mental load to denote the aspects of a task (and environment) that influence its cognitive load, i.e., the workload on the human cognitive system (Paas and Van Merriënboer, 1994).…”

“…Mental load will increase with the size and complexity of the code change, and therefore review performance should be lower for more complex code changes. Furthermore, our recent re-search (Baum et al, 2017b) proposes that automatic improvement of the order in which the parts of the code change (e.g., changed methods or changed files) are presented to the reviewer can reduce mental load and therefore increase review performance.…”

Test-Driven Code Review: An Empirical Study

“…Research on the ordering of code changes is also related to TDR. In particular, Baum et al argued that an optimal ordering of code changes would help reviewers by reducing the cognitive load and improving the alignment with their cognitive processes [10], even though they made no explicit reference to ordering tests. This may give theoretical value to the TDR practice.…”