Will They Like This? Evaluating Code Contributions with Language Models

Abstract

Popular open-source software projects receive and review contributions from a diverse array of developers, many of whom have little to no prior involvement with the project. A recent survey reported that reviewers consider conformance to the project's code style to be one of the top priorities when evaluating code contributions on Github. We propose to quantitatively evaluate the existence and effects of this phenomenon. To this aim we use language models, which were shown to accurately capture stylistic aspects of code. We find that rejected changesets do contain code significantly less similar to the project than accepted ones; furthermore, the less similar changesets are more likely to be subject to thorough review. Armed with these results we further investigate whether new contributors learn to conform to the project style and find that experience is positively correlated with conformance to the project's code style.

Effective assignment and assistance to software developers and reviewers

“…For selecting features we used some features from [17,19] and [44] and we extends the features. Features such as severity, owner bug assignee match, file size, entropy score, number of assigned and contributing reviewers, and assigned reviewer's reputation are not investigated at [17,19].…”

“…Features such as severity, owner bug assignee match, file size, entropy score, number of assigned and contributing reviewers, and assigned reviewer's reputation are not investigated at [17,19]. The only feature which has been investigated in [44] is the entropy score.…”

“…It shows how many times the patch owner had to revise the patch (probably to address the review Entropy Score: This feature is based on previous work of Hellendoorn et al [44]. The study proposed that the project's coding style is one of the important features which reviewers consider.…”

“…Hellendoorn et al [44] states that to judge the similarity of a sequence of tokens with respect to a corpus, a language model assigns it a probability by counting the relative frequency of the sequence of tokens in the corpus. In the natural language setting, these models are used for tasks such as speech recognition, machine translation, and spelling correction.…”

“…Clearly, MCR is quite human intensive, which may require a substantial amount of effort [17], [50], [44], [19]. Also, the predictive recommendation and assessment of whether a patch will be eventually accepted or not as soon as it is submitted for MCR were not investigated previously.…”