The role of software in science: a knowledge graph-based analysis of software mentions in PubMed Central

Abstract

Science across all disciplines has become increasingly data-driven, leading to additional needs with respect to software for collecting, processing and analysing data. Thus, transparency about software used as part of the scientific process is crucial to understand provenance of individual research data and insights, is a prerequisite for reproducibility and can enable macro-analysis of the evolution of scientific methods over time. However, missing rigor in software citation practices renders the automated detection and disambiguation of software mentions a challenging problem. In this work, we provide a large-scale analysis of software usage and citation practices facilitated through an unprecedented knowledge graph of software mentions and affiliated metadata generated through supervised information extraction models trained on a unique gold standard corpus and applied to more than 3 million scientific articles. Our information extraction approach distinguishes different types of software and mentions, disambiguates mentions and outperforms the state-of-the-art significantly, leading to the most comprehensive corpus of 11.8 M software mentions that are described through a knowledge graph consisting of more than 300 M triples. Our analysis provides insights into the evolution of software usage and citation patterns across various fields, ranks of journals, and impact of publications. Whereas, to the best of our knowledge, this is the most comprehensive analysis of software use and citation at the time, all data and models are shared publicly to facilitate further research into scientific use and citation of software.

FAIRsoft - A practical implementation of FAIR principles for research software

“…Indeed, this effort is made in the context of the Life Sciences. However, similar trends have been described across other scientific disciplines (Schindler et al ., 2022). Such trends highlight the need of a concerted and transversal effort for realising the value of research software to the scientific endeavour and advancing on better practices for its development and sustainability over time.…”

“…Even when a piece of research software is explicitly mentioned in a research publication, the specific version used is still sometimes left unspecified (Howison and Bullard, 2016), severely hindering attempts to reuse it or to reproduce published results. For this reason, "Specificity" is one of the Software Citation Principles (Smith et al, 2016). The usage of a versioning scheme in terms of version identification (F1.2: A scheme is used to uniquely and properly identify the software version) or provenance (R4.1: The software follows a version-control system) (Figure 3D) emerges as a key factor to allow the proper use of the different versions of an evolving artefact such as a piece of software and, thus, guaranteeing both its Findability and Reusability.…”

“…Computational tools are increasingly becoming constitutive parts of scientific research, from experimentation and data collection, to the dissemination and storage of results. This dependence on computational technologies is especially strong in research based on computational simulations and data driven Science (Schindler et al, 2022). Computational simulations allow modelling phenomena that cannot be reproduced empirically for its analysis, constituting the first epistemological shift caused by computational technologies in Science (Hey, 2012) .…”

Understanding progress in software citation: a study of software citation in the CORD-19 corpus

“…, Schindler et al, 2021 ; Du et al, 2021a ). Supervised ML-based software extraction over large collections with acceptable computational performance is now possible ( Lopez et al, 2021a ; Lopez et al, 2021b ; Schindler et al, 2022 ; Wade & Williams, 2021 ). Community efforts titled “Habeus Corpus” was launched and continue to investigate these emerging collections of extracted software mentions ( Habeas Corpus, 2021 ).…”

A large dataset of software mentions in the biomedical literature

“…These datasets are based on limited samples of scholarly articles. A paper based on a similar approach [26] was published while this preprint was in preparation. Similarly to our work, it is based on Pubmed Central Open Access dataset.…”