A Large-Scale Study on Source Code Reviewer Recommendation

Abstract

Context: Software code reviews are an important part of the development process, leading to better software quality and reduced overall costs. However, finding appropriate code reviewers is a complex and time-consuming task. Goals: In this paper, we propose a large-scale study to compare performance of two main source code reviewer recommendation algorithms (RevFinder and a Naïve Bayes-based approach) in identifying the best code reviewers for opened pull requests. Method: We mined data from Github and Gerrit repositories, building a large dataset of 51 projects, with more than 293K pull requests analyzed, 180K owners and 157K reviewers. Results: Based on the large analysis, we can state that i) no model can be generalized as best for all projects, ii) the usage of a different repository (Gerrit, GitHub) can have impact on the the recommendation results, iii) exploiting sub-projects information available in Gerrit can improve the recommendation results.• provision of a large dataset of 51 projects mined from Gerrit (14) and Github (37) with 293,337 total pull requests analyzed, considering 180,111 owners and 157,885 reviewers. The dataset is available on Figshare [24]. • comparison of the results in the two repositories (Gerrit, GitHub) using both RevFinder and Naïve Bayes-based approaches in the context of the 51-projects dataset.The article is structured as follows. In section II, we propose the background on several algorithms for code reviewer recommendation. In section III we have the experimental study design, with research questions, context, data analysis and replicability information. In section IV, we answer the research questions with discussions and threats to validity. Section V proposes the related works and section VI the conclusions.

Reviewer Recommendation using Software Artifact Traceability Graphs

“…We also searched for publicly available datasets for further comparison. Lipcak and Rossi [12] review different approaches on code reviewer recommendation and create a dataset publicly available [12] Naive-Bayes --Code RevFinder [22] File path similarity + -Code Correct [18] Developer experience -+ Code RevRec [26] Hybrid of information retrieval and file location --Code Jeong et al [6] Bayesian network -+ Code cHRev [29] Expertise model --Code Developer-Source code graph [11] Random walk algorithm --Code TIE [24] Text and File Location Analyses --Code CoreDevRec [7] Support vector machine --Code RSTrace Know-about metric + + Any for comparison with other approaches. The dataset includes code review information about 51 projects (37 from GitHub and 14 from Gerrit) and it is available on GitHub.…”

“…In this subsection, results from these approaches will be discussed and compared with RSTrace. We implemented the Naive-Bayes algorithm that is described by Lipcak and Rossi [12]. The results of the comparison between RSTrace and Naive-Bayes is shown in Figure 8.…”

“…We run RSTrace on 37 projects, and the results of top-5 accuracy and MRR scores per each project is shown in Figure 9. According to [12], RevFinder reaches MRR mean of 55 and Naive-Bayes with MRR mean of 61.62. According to the results, RSTrace reaches an MRR mean of 61, which is compatible with RevFinder and Naive-Bayes.…”

“…• Running RSTrace vs. Naive-Bayes algorithm in Qt 3D Studio dataset. • Running RSTrace vs Naive-Bayes and RevFinder in 37 opensource projects described in [12] Results: In Qt 3D Studio data, our approach has an improvement of 49% for top-1, 15,67% for top-2, and 8,59% for top-3 respectively according to the baseline approach. RSTrace also has a 28% improvement in MRR scores compare to Naive-Bayes algorithm.…”

Core-reviewer recommendation based on Pull Request topic model and collaborator social network

“…Yu et al (2014) proposed a reviewer recommendation algorithm based on the review network. Lipcak et al (2018) conducted big data experiments on various methods above and found that the results of these methods for large-scale projects are not very satisfactory, but they have a good effect on medium-scale projects. Xia et al (2017) proposed a recommendation algorithm that considers implicit relations and neighborhood models.…”