Curating GitHub for engineered software projects

Abstract

Software forges like GitHub host millions of repositories. Software engineering researchers have been able to take advantage of such a large corpora of potential study subjects with the help of tools like GHTorrent and Boa. However, the simplicity in querying comes with a caveat: there are limited means of separating the signal (e.g. repositories containing engineered software projects) from the noise (e.g. repositories containing home work assignments). The proportion of noise in a random sample of repositories could skew the study and may lead to researchers reaching unrealistic, potentially inaccurate, conclusions. We argue that it is imperative to have the ability to sieve out the noise in such large repository forges. We propose a framework, and present a reference implementation of the framework as a tool called reaper, to enable researchers to select GitHub repositories that contain evidence of an engineered software project. We identify software engineering practices (called dimensions) and propose means for validating their existence in a GitHub repository. We used reaper to measure the dimensions of 1,994,977 GitHub repositories. We then used the data set train classifiers capable of predicting if a given GitHub repository contains an engineered software project. The performance of the classifiers was evaluated using a set of 200 repositories with known ground truth classification. We also compared the performance of the classifiers to other approaches to classification (e.g. number of GitHub Stargazers) and found our classifiers to outperform existing approaches. We found stargazers-based classifier to exhibit high precision (96%) but an inversely proportional recall (27%). On the other hand, our best classifier exhibited a high precision (82%) and a high recall (83%). The stargazer-based criteria offers precision but fails to recall a significant potion of the population.

Understanding How GitHub Supports Curation Repositories

“…The research focus of Munaiah et al (2017) was different from the one of this paper, since they developed a set of metrics and curated software repositories from GitHub according to these metrics, which is a method of algorithmic curating [13]. In contrast, in this study, we investigate the curation behavior happening on GitHub that appropriates GitHub repositories for curation purposes, which is a different category of curation in the software practice.…”

“…A recent study by Munaiah et al (2017) developed a tool, called reaper, that evaluates a GitHub repository from eight dimensions to determine whether a repository is an engineered software project and to identify software projects that conform to the dimensions within a sample of 1,994,977 GitHub repositories [13]. The research focus of Munaiah et al (2017) was different from the one of this paper, since they developed a set of metrics and curated software repositories from GitHub according to these metrics, which is a method of algorithmic curating [13].…”

Automatic Generation of Pull Request Descriptions

“…To collect PR data from GitHub, we first used the RepoReapers framework [21] to select engineered software projects. We obtained all 95,804 Java repositories that had been classified as containing engineered software projects by RepoReapers's Random Forest classification and retrieved the number of merged PRs for each repository.…”

Do as I Do, Not as I Say: Do Contribution Guidelines Match the GitHub Contribution Process?

“…In order to filter the large dataset provided by GHTorrent (about 37 million projects), we followed criteria laid out by Vasilescu et al [7], Tsay et al [16], and Munaiah et al [17]. The combination of the criteria from the previously mentioned literature resulted in the following filters:…”

“…We focused on projects where a contributorparticularly one who has no write privileges to the source repository-has access to the build results. • Exclude projects that have less than three unique contributors: This is an indicator of the project having a tightly-knit community of developers that are actively collaborating but are less inclined to accept external contribution, as discussed by Munaiah et al [17]. • Exclude projects that do not have at least one recently merged pull request: According to Kalliamvakou et al [18], having a pull request does not indicate that it was merged.…”