A Complete Set of Related Git Repositories Identified via Community Detection Approaches Based on Shared Commits

Abstract

In order to understand the state and evolution of the entirety of open source software we need to get a handle on the set of distinct software projects. Most of open source projects presently utilize Git, which is a distributed version control system allowing easy creation of clones and resulting in numerous repositories that are almost entirely based on some parent repository from which they were cloned. Git commits are unlikely to get produce and represent a way to group cloned repositories. We use World of Code infrastructure containing approximately 2B commits and 100M repositories to create and share such a map. We discover that the largest group contains almost 14M repositories most of which are unrelated to each other. As it turns out, the developers can push git object to an arbitrary repository or pull objects from unrelated repositories, thus linking unrelated repositories. To address this, we apply Louvain community detection algorithm to this very large graph consisting of links between commits and projects. The approach successfully reduces the size of the megacluster with the largest group of highly interconnected projects containing under 400K repositories. We expect that the resulting map of related projects as well as tools and methods to handle the very large graph will serve as a reference set for mining software projects and other applications. Further work is needed to determine different types of relationships among projects induced by shared commits and other relationships, for example, by shared source code or similar filenames.

On the analysis of non-coding roles in open source development

“…This is part of our future work, together with the replication of the study on other sets of projects. In this sense, we are especially interested in studying how these observations evolve when moving to project ecosystems (Blincoe et al 2015;Mockus et al 2020) instead of single projects.…”

Revisiting Dockerfiles in Open Source Software Over Time

“…The latest S version contains 9,192,143,411 unique blobs, 2,326,066,436 commits, and 135,162,320 distinct repositories collected from open source communities including GitHub, Bitbucket, and GitLab identified on August 28, 2020 and retrieved by September 18, 2020 [12]. A distinct repository is determined as the "most central" repository to represent a group of repositories found with the Louvain community detection algorithm [13]. By using distinct repositories, many cloned projects (forks) can be avoided when performing an analysis.…”

Representation of Developer Expertise in Open Source Software

“…To address this, we use the dataset published in [46], which applies the Louvain community detection algorithm to a massive graph consisting of links between commits and projects in WoC (because two projects are highly unlikely to share the same exact commit unless they are clones). We leverage that work to combine commits from the forked projects and ensure that we do not count the same projectrelated information multiple times due to these forks/clones.…”