Nine Best Practices for Research Software Registries and Repositories: A Concise Guide

Registries, Task Force on Best Practices for Software; Monteil, Alain; González-Beltrán, Alejandra; Ioannidis, Alexandros; Allen, Alice; Lee, Allen; Bandrowski, Anita; Wilson, Bruce E.; Mecum, Bryce; Du, Cai; Robinson, Carly D.; Garijo, Daniel; Katz, Daniel S.; Long, D.G.; Milliken, Genevieve; Ménager, Hervé; Hausman, Jessica; Spaaks, Jurriaan H.; Fenlon, Katrina; Vanderbilt, Kristin; Hwang, Lorraine; Davis, Lynn E.; Fenner, Martin; Crusoe, Michael R.; Hucka, Michael; Wu, Mingfang; Hong, Neil Chue; Teuben, Peter; Stall, Shelley; Druskat, Stephan; Carnevale, Ted; Morrell, Thomas E.

doi:10.48550/arxiv.2012.13117

Cited by 3 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A. Software Package Reuse Software package registries store versioned packaged software, associated metadata, documentation, and configurations [20]. Similarly, deep learning (DL) model registries distribute PTMs with metadata, a model card (i.e., documentation), relevant configurations, and versions of pre-trained weights [21].…”

Section: Background and Related Workmentioning

confidence: 99%

An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

Jiang¹,

Synovic²,

Hyatt³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are being adopted as components in software systems. Creating and specializing DNNs from scratch has grown increasingly difficult as stateof-the-art architectures grow more complex. Following the path of traditional software engineering, machine learning engineers have begun to reuse large-scale pre-trained models (PTMs) and fine-tune these models for downstream tasks. Prior works have studied reuse practices for traditional software packages to guide software engineers towards better package maintenance and dependency management. We lack a similar foundation of knowledge to guide behaviors in pre-trained model ecosystems.In this work, we present the first empirical investigation of PTM reuse. We interviewed 12 practitioners from the most popular PTM ecosystem, Hugging Face, to learn the practices and challenges of PTM reuse. From this data, we model the decision-making process for PTM reuse. Based on the identified practices, we describe useful attributes for model reuse, including provenance, reproducibility, and portability. Three challenges for PTM reuse are missing attributes, discrepancies between claimed and actual performance, and model risks. We substantiate these identified challenges with systematic measurements in the Hugging Face ecosystem. Our work informs future directions on optimizing deep learning ecosystems by automated measuring useful attributes and potential attacks, and envision future research on infrastructure and standardization for model registries.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

Jiang¹,

Synovic²,

Hyatt³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Nine best practices for research software registries and repositories

Garijo

Ménager

Hwang

et al. 2022

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Scientific software registries and repositories improve software findability and research transparency, provide information for software citations, and foster preservation of computational methods in a wide range of disciplines. Registries and repositories play a critical role by supporting research reproducibility and replicability, but developing them takes effort and few guidelines are available to help prospective creators of these resources. To address this need, the FORCE11 Software Citation Implementation Working Group convened a Task Force to distill the experiences of the managers of existing resources in setting expectations for all stakeholders. In this article, we describe the resultant best practices which include defining the scope, policies, and rules that govern individual registries and repositories, along with the background, examples, and collaborative work that went into their development. We believe that establishing specific policies such as those presented here will help other scientific software registries and repositories better serve their users and their disciplines.

show abstract

Special issue on software citation, indexing, and discoverability

Katz,

Chue Hong

2024

PeerJ Computer Science

Self Cite

View full text Add to dashboard Cite

Software plays a fundamental role in research as a tool, an output, or even as an object of study. This special issue on software citation, indexing, and discoverability brings together five papers examining different aspects of how the use of software is recorded and made available to others. It describes new work on datasets that enable large-scale analysis of the evolution of software usage and citation, that presents evidence of increased citation rates when software artifacts are released, that provides guidance for registries and repositories to support software citation and findability, and that shows there are still barriers to improving and formalising software citation and publication practice. As the use of software increases further, driven by modern research methods, addressing the barriers to software citation and discoverability will encourage greater sharing and reuse of software, in turn enabling research progress.

show abstract

Nine Best Practices for Research Software Registries and Repositories: A Concise Guide

Cited by 3 publications

References 0 publications

An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registry

Nine best practices for research software registries and repositories

Special issue on software citation, indexing, and discoverability

Contact Info

Product

Resources

About