Investigating the Validity of Ground Truth in Code Reviewer Recommendation Studies

Abstract

Background: Selecting the ideal code reviewer in modern code review is a crucial first step to perform effective code reviews. There are several algorithms proposed in the literature for recommending the ideal code reviewer for a given pull request. The success of these code reviewer recommendation algorithms is measured by comparing the recommended reviewers with the ground truth that is the assigned reviewers selected in real life. However, in practice, the assigned reviewer may not be the ideal reviewer for a given pull request. Aims: In this study, we investigate the validity of ground truth data in code reviewer recommendation studies. Method: By conducting an informal literature review, we compared the reviewer selection heuristics in real life and the algorithms used in recommendation models. We further support our claims by using empirical data from code reviewer recommendation studies. Results: By literature review, and accompanying empirical data, we show that ground truth data used in code reviewer recommendation studies is potentially problematic. This reduces the validity of the code reviewer datasets and the reviewer recommendation studies. Conclusion: We demonstrated the cases where the ground truth in code reviewer recommendation studies are invalid and discussed the potential solutions to address this issue.

Using Large-scale Heterogeneous Graph Representation Learning for Code Review Recommendations at Microsoft

“…To assess the value of Coral empircally, we pose three research questions: The vast majority of code reviewer recommendation approaches are evaluated by comparing recommendations from the tool with historical code reviews and examining how often the recommended reviewers were the actual reviewers [13]. In line with this accepted practice, RQ1 asks how often the network is able to recommend the reviewers that the authors added.…”

“…In line with this accepted practice, RQ1 asks how often the network is able to recommend the reviewers that the authors added. However, as Dougan et al point out, there is an underlying (and often unstated) assumption that these are the correct reviewers * [13]. To address this flawed assumption and pursue a more complete ground truth, we reach out to the reviewers recommended by Coral that were not recommended by a rule-based model.…”

“…Rahman et al [25] propose CORRECT an approach to recommend reviewers based on their history across all of GitHub as well as their experience certain specialized technologies associated with a pull request. Dougan et al [13] investigated the problem of ground truth in reviewer recommendation systems. They point out that many tools are trained and evaluated on historical code reviews and rely on an (often unstated) assumption that the selected reviewers were the correct reviewers.…”

“…In this context, the "correct" recommendation is defined as the recommended reviewer being invited to the pull request. While the author's actions may not actually reflect the ground truth of who is best able to review the change, most prior work in code reviewer recommendation evaluates recommenders in this way (see [13] for a thorough discussion of this) and so we follow suit here. Table 4 shows the accuracy and MRR for Coral across all 254K (pull request-reviewer) pairs.…”

Towards a taxonomy of code review smells

“…Selecting a proper reviewer is an important initial step for effective code reviews [58]. Although getting a file reviewed by an expert or a senior developer seems to be an advantage, the code review activity must be balanced within the team to increase the shared code ownership [2].…”