Workload-aware reviewer recommendation using a multi-objective search-based approach

Abstract

Reviewer recommendation approaches have been proposed to provide automated support in finding suitable reviewers to review a given patch. However, they mainly focused on reviewer experience, and did not take into account the review workload, which is another important factor for a reviewer to decide if they will accept a review invitation. We set out to empirically investigate the feasibility of automatically recommending reviewers while considering the review workload amongst other factors. We develop a novel approach that leverages a multi-objective meta-heuristic algorithm to search for reviewers guided by two objectives, i.e., (1) maximizing the chance of participating in a review, and (2) minimizing the skewness of the review workload distribution among reviewers. Through an empirical study of 230,090 patches with 7,431 reviewers spread across four open source projects, we find that our approach can recommend reviewers who are potentially suitable for a newly-submitted patch with 19% -260% higher F-measure than the five benchmarks. Our empirical results demonstrate that the review workload and other important information should be taken into consideration in finding reviewers who are potentially suitable for a newly-submitted patch. In addition, the results show the effectiveness of realizing this approach using a multi-objective search-based approach. CCS CONCEPTS• Software and its engineering → Search-based software engineering; Collaboration in software development.

Modern Code Reviews—Survey of Literature and Practice

“…Another interesting direction is to focus recommend reviewers that will ensure code base knowledge distribution [86,176,207]. Finally, some studies have included balancing review workload as an objective [43,49,86,230] In relation to how the predictors are used to recommend code reviewers, many employ traditional approaches (e.g., cosine similarity), while some use machine learning techniques, such as Random Forest [92], Naive Bayes [92,235], Support Vector Machines [144,276], Collaborative Filtering [87,230], Deep Neural Networks [222,274], or model reviewer recommendation as an optimization problem [43,86,187,207,211].…”

“…Another interesting direction is to focus recommend reviewers that will ensure code base knowledge distribution [86,176,207]. Finally, some studies have included balancing review workload as an objective [43,49,86,230].…”

An Exploration of Cross-Patch Collaborations via Patch Linkage in OpenStack

“…Hannebauer et al [22] recommended code reviewers based on their expertise. Most recently, Al-Zubaidi et al [23] developed a novel approach that leverages a multiobjective meta-heuristic algorithm to search for reviewers guided by two objectives.…”

Code Reviewer Recommendation for Architecture Violations: An Exploratory Study

“…In terms of the incremental sampling, we set four steps in this work, and we took 10% of the new sample as the validation set in each step. In terms of the fixed sampling, we employed a fixed percentage of the test set by randomly sampling with 10% in previous studies (e.g., [1]) of the dataset of the four projects.…”