Curating GitHub for engineered software projects

Abstract

12Software forges like GitHub host millions of repositories. Software engineering researchers have been able to take advantage of such a large corpora of potential study subjects with the help of tools like GHTorrent and Boa. However, the simplicity in querying comes with a caveat: there are limited means of separating the signal (e.g. repositories containing engineered software projects) from the noise (e.g. repositories containing home work assignments). The proportion of noise in a random sample of repositories could skew the study and may lead to researchers reaching unrealistic, potentially inaccurate, conclusions. We argue that it is imperative to have the ability to sieve out the noise in such large repository forges. We propose a framework, and present a reference implementation of the framework as a tool called reaper, to enable researchers to select GitHub repositories that contain evidence of an engineered software project. We identify software engineering practices (called dimensions) and propose means for validating their existence in a GitHub repository. We used reaper to measure the dimensions of 1,994,977 GitHub repositories. We then used the data set train classifiers capable of predicting if a given GitHub repository contains an engineered software project. The performance of the classifiers was evaluated using a set of 200 repositories with known ground truth classification. We also compared the performance of the classifiers to other approaches to classification (e.g. number of GitHub Stargazers) and found our classifiers to outperform existing approaches. We found stargazers-based classifier to exhibit high precision (96%) but an inversely proportional recall (27%). On the other hand, our best classifier exhibited a high precision (82%) and a high recall (83%). The stargazer-based criteria offers precision but fails to recall a significant potion of the population.

The Silent Helper: The Impact of Continuous Integration on Code Reviews

“…Secondly, the criteria used to filter out inactive, or noncollaborative projects may introduce bias; Work by Munaiah et al [55] has shown that the recall of such filtering techniques can be 30% or lower. Therefore, there could be a class of smaller yet active and collaborative open-source software projects which we are not represented in our study.…”

Discovering community patterns in open-source: a systematic approach and its evaluation

“…Finally, we have manually inspected the selected projects and made sure that all of them are real projects (rather than student projects, assignments, etc. ), as suggested by recent work (Munaiah et al 2017). The specific query employed for the selection of the subject projects was done on April 2017 and can be found in our on-line appendix ).…”

World of Code: An Infrastructure for Mining the Universe of Open Source VCS Data

“…WoC can be extended with other approaches to segment projects 17 . For example, identification of projects with sound software engineering practices [58] relies on a combination of factors easily obtainable in WoC, such as history, license, and unit tests.…”