Sourcerer: mining and searching internet-scale software repositories

Abstract

Large repositories of source code available over the Internet, or within large organizations, create new challenges and opportunities for data mining and statistical machine learning. Here we first develop Sourcerer, an infrastructure for the automated crawling, parsing, fingerprinting, and database storage of open source software on an Internet-scale. In one experiment, we gather 4,632 Java projects from SourceForge and Apache totaling over 38 million lines of code from 9,250 developers. Simple statistical analyses of the data first reveal robust power-law behavior for package, method call, and lexical containment distributions. We then develop and apply unsupervised, probabilistic, topic and author-topic (AT) models to automatically 123 Mining and searching internet-scale software repositories 301 discover the topics embedded in the code and extract topic-word, document-topic, and AT distributions. In addition to serving as a convenient summary for program function and developer activities, these and other related distributions provide a statistical and information-theoretic basis for quantifying and analyzing source file similarity, developer similarity and competence, topic scattering, and document tangling, with direct applications to software engineering an software development staffing. Finally, by combining software textual content with structural information captured by our CodeRank approach, we are able to significantly improve software retrieval performance, increasing the area under the curve (AUC) retrieval metric to 0.92-roughly 10-30% better than previous approaches based on text alone. A prototype of the system is available at: http://sourcerer.ics.uci.edu.

Mining source code repositories at massive scale using language modeling

“…However, this work does not quantify how identifier properties vary, since it ignores variable and type names. Search of code at "internet-scale" was introduced by Linstead et al [8]. Another GitHub dataset, GHTorrent [9] has a different goal compared to our corpus, excluding source code and focusing on users, pull requests and all the issues surrounding social coding.…”

HAPA: Harvester and Pedagogical Agents in E-learning Environments

“…In order to improve the Classifier's performance, more intelligent source code classification techniques will be implemented in the future (e.g. [12]). …”

Stakeholders' social interaction in requirements engineering of open source software

“…Topic modeling has recently been used in several research areas of software engineering, such as mining software repositories (MSR) [108,109,188], requirements traceability [7], and software evolution [111]. Linstead et al [109] applied LDA topic modeling technique on the source code of different versions in order to analyze software evolution.…”

“…Linstead et al [109] applied LDA topic modeling technique on the source code of different versions in order to analyze software evolution. Linstead and colleagues [108] further used topic modeling on Internet-scale software repositories, and summarized program function and developer activities by extracting topic-word and author-topic distributions. The use of topic modeling over source code has been validated and it has been found that the evolution of source code topics is indeed caused by actual change activities in the code [188].…”