Confusion Detection in Code Reviews

Abstract

Code reviews are an important mechanism for assuring quality of source code changes. Reviewers can either add general comments pertaining to the entire change or pinpoint concerns or shortcomings about a specific part of the change using inline comments. Recent studies show that reviewers often do not understand the change being reviewed and its context. Our ultimate goal is to identify the factors that confuse code reviewers and understand how confusion impacts the efficiency and effectiveness of code review(er)s. As the first step towards this goal we focus on the identification of confusion in developers' comments. Based on an existing theoretical framework categorizing expressions of confusion, we manually classify 800 comments from code reviews of the Android project. We observe that confusion can be reasonably well-identified by humans: raters achieve moderate agreement (Fleiss' kappa 0.59 for the general comments and 0.49 for the inline ones). Then, for each kind of comment we build a series of automatic classifiers that, depending on the goals of the further analysis, can be trained to achieve high precision (0.875 for the general comments and 0.615 for the inline ones), high recall (0.944 for the general comments and 0.988 for the inline ones), or substantial precision and recall (0.696 and 0.542 for the general comments and 0.434 and 0.583 for the inline ones, respectively). These results motivate further research on the impact of confusion on the code review process. Moreover, other researchers can employ the proposed classifiers to analyze confusion in other contexts where software development-related discussions occur, such as mailing lists. Index Terms-code review; confusion; machine learning.

The Silent Helper: The Impact of Continuous Integration on Code Reviews

“…To better understand how integrators review code Yu et al investigated the latency of open-source code reviews, coming to the conclusion that many process related features impact the latency of code reviews [29]. Ebert et al attempted to find expressions of confusion in code reviews, to further understand how confusion impacts code reviews [30]. Beller et al investigate what types of changes are made during a code review, finding that most changes are made to improve evolvability [31].…”

Associating working memory capacity and code change ordering with code review performance

“…When studying the cognitive level of think-aloud statements during reviews, McMeekin et al (2009) found that more structured techniques lead to higher cognition levels. Recent work by Ebert et al (2017) tries to measure signs of confusion from remarks written by the reviewers. The model of human and computer as a joint cognitive system (Dowell and Long, 1998), also called distributed cognition, has been proposed by Walenstein to study cognitive load in software development tools (Walenstein, 2003(Walenstein, , 2002.…”

Communicative Intention in Code Review Questions

“…We conduct an explanatory case study [6] on Android because it is a large and well-known open source ecosystem that adopts a rigorous code review process using Gerrit. We extracted our annotation sample from the dataset of inline comments we previously collected from the entire Android ecosystem [17]. Our annotation sample contains 499 questions extracted from randomly selected 399 inline comments (corresponding to the confidence interval of less than 5% and confidence level of 95% from a population of 10,965 questions).…”