World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data

Ma, Yuxing; Dey, Tapajit; Bogart, Christopher; Amreen, Sadika; Valiev, Marat; Tutko, Adam; Kennard, David; Zaretzki, Russell; Mockus, Audris

doi:10.1007/s10664-020-09905-9

Cited by 30 publications

(12 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…World of Code [14] is a large dataset and analysis infrastructure, available to researchers to mine public code. It is larger than our initial data source and can be used in conjunction with this dataset to find additional origins/occurrences of licenses blobs of interest.…”

Section: Related Workmentioning

confidence: 99%

A Large-scale Dataset of (Open Source) License Text Variants

Zacchiroli

2022

Preprint

View full text Add to dashboard Cite

We introduce a large-scale dataset of the complete texts of free/open source software (FOSS) license variants. To assemble it we have collected from the Software Heritage archive-the largest publicly available archive of FOSS source code with accompanying development history-all versions of files whose names are commonly used to convey licensing terms to software users and developers.The dataset consists of 6.5 million unique license files that can be used to conduct empirical studies on open source licensing, training of automated license classifiers, natural language processing (NLP) analyses of legal texts, as well as historical and phylogenetic studies on FOSS licensing.Additional metadata about shipped license files are also provided, making the dataset ready to use in various contexts; they include: file length measures, detected MIME type, detected SPDX license (using ScanCode), example origin (e.g., GitHub repository), oldest public commit in which the license appeared.The dataset is released as open data as an archive file containing all deduplicated license files, plus several portable CSV files for metadata, referencing files via cryptographic checksums.

show abstract

Section: Related Workmentioning

confidence: 99%

A Large-scale Dataset of (Open Source) License Text Variants

Zacchiroli

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Other tools offering an infrastructure framework include Crossflow 4 [7], SmartSHARK 5 [15], and World of Code [9,10]. CrossFlow is a domain specific language and framework that offers scalability for MSR tasks; SmartSHARK aggregates data from different sources in a harmonized schema.…”

Section: Related Workmentioning

confidence: 99%

“…One goal is to quantify and monitor the quality of the developed software employing software metrics [8], another is to explore new ways of making sense of the data. However, traversing and gathering the vast amount of data poses a challenge [4,[9][10][11].…”

Section: Introductionmentioning

confidence: 99%

Tooling for Time- and Space-efficient git Repository Mining

Heseding,

Scheibel,

Döllner

2022

Preprint

View full text Add to dashboard Cite

Software projects under version control grow with each commit, accumulating up to hundreds of thousands of commits per repository. Especially for such large projects, the traversal of a repository and data extraction for static source code analysis poses a trade-off between granularity and speed.We showcase the command-line tool pyrepositoryminer that combines a set of optimization approaches for efficient traversal and data extraction from git repositories while being adaptable to third-party and custom software metrics and data extractions. The tool is written in Python and combines bare repository access, in-memory storage, parallelization, caching, change-based analysis, and optimized communication between the traversal and custom data extraction components. The tool allows for both metrics written in Python and external programs for data extraction. A single-thread performance evaluation based on a basic mining use case shows a mean speedup of 15.6× to other freely available tools across four mid-sized open source projects. A multi-threaded execution allows for load distribution among cores and, thus, a mean speedup up to 86.9× using 12 threads. CCS CONCEPTS• Software and its engineering → Software libraries and repositories; Software maintenance tools; Software version control.

show abstract

“…The challenge with using this dataset for training purposes is that it contains only 1471 vulnerable functions and 1320 vulnerable files, which is rather small for training a deep learning model. Software Repositories and Repository Mining Frameworks Ma et al [30] present World of Code (WoC), a large and frequently updated collection of git-based version control data. The data is indexed three storage abstractions and arranged in four layers.…”

Section: Related Workmentioning

confidence: 99%

CVEfixes: automated collection of vulnerabilities and their fixes from open-source software

Bhandari

Naseer

Moonen

2021

Proceedings of the 17th International Conference on Predictive Models and Data Analytics in Software Engineering

View full text Add to dashboard Cite

Data-driven research on the automated discovery and repair of security vulnerabilities in source code requires comprehensive datasets of real-life vulnerable code and their fixes. To assist in such research, we propose a method to automatically collect and curate a comprehensive vulnerability dataset from Common Vulnerabilities and Exposures (CVE) records in the public National Vulnerability Database (NVD). We implement our approach in a fully automated dataset collection tool and share an initial release of the resulting vulnerability dataset named CVEfixes.The CVEfixes collection tool automatically fetches all available CVE records from the NVD, gathers the vulnerable code and corresponding fixes from associated open-source repositories, and organizes the collected information in a relational database. Moreover, the dataset is enriched with meta-data such as programming language, and detailed code and security metrics at five levels of abstraction. The collection can easily be repeated to keep up-todate with newly discovered or patched vulnerabilities. The initial release of CVEfixes spans all published CVEs up to 9 June 2021, covering 5365 CVE records for 1754 open-source projects that were addressed in a total of 5495 vulnerability fixing commits.CVEfixes supports various types of data-driven software security research, such as vulnerability prediction, vulnerability classification, vulnerability severity prediction, analysis of vulnerabilityrelated code changes, and automated vulnerability repair. CCS CONCEPTS• Security and privacy → Software and application security; Vulnerability management; • Software and its engineering → Software defect analysis; Software libraries and repositories.

show abstract

World of code: enabling a research workflow for mining and analyzing the universe of open source VCS data

Cited by 30 publications

References 57 publications

A Large-scale Dataset of (Open Source) License Text Variants

A Large-scale Dataset of (Open Source) License Text Variants

Tooling for Time- and Space-efficient git Repository Mining

CVEfixes: automated collection of vulnerabilities and their fixes from open-source software

Contact Info

Product

Resources

About