Assessing and tracing the outcomes and impact of research infrastructures

Abstract

Recent policy shifts on the part of funding agencies and journal publishers are causing changes in the acknowledgment and citation behaviors of scholars. A growing emphasis on open science and reproducibility is changing how authors cite and acknowledge "research infrastructures"-entities that are used as inputs to or as underlying foundations for scholarly research, including data sets, software packages, computational models, observational platforms, and computing facilities. At the same time, stakeholder interest in quantitative understanding of impact is spurring increased collection and analysis of metrics related to use of research infrastructures. This article reviews work spanning several decades on tracing and assessing the outcomes and impacts from these kinds of research infrastructures. We discuss how research infrastructures are identified and referenced by scholars in the research literature and how those references are being collected and analyzed for the purposes of evaluating impact. Synthesizing common features of a wide range of studies, we identify notable challenges that impede the analysis of impact metrics for research infrastructures and outline key open research questions that can guide future research and applications related to such metrics.

The Oklahoma Mesonet: A Pilot Study of Environmental Sensor Data Citations

“…In an extensive review of research efforts similar to the Oklahoma Mesonet pilot project described here, Mayernik et al (2017Mayernik et al ( : 1354 have suggested that practical, in-depth studies of specific 'research infrastructures' that routinely provide significant research assets, such as datasets and datastreams, also offer unique Utilization (Kenkel & Norris 1995: 365) The two objectives of the study were to estimate average willingness to pay and to determine the characteristics of producers who would pay to access and use mesoscale weather information. To this end, two maximum likelihood models were estimated: one for the raw [Oklahoma] Mesonet data and one for the raw data/value-added information combination .…”

“…Although the Oklahoma Mesonet has been routinely compiling a bibliography of publications pertinent to its efforts since its inception, the study described here is an initial attempt toward a formal analysis of the peerreviewed scientific publications, their uses of the Oklahoma Mesonet data, and toward further efforts to understand and facilitate such Mesonet data usage. Mayernik et al (2017Mayernik et al ( : 1344 assert that 'Manual processes for collecting publications that cite data sets . .…”

Theory and practice of data citation

“…5. Data impact: Citing data allows us to define new usage metrics for determining the impact of data; data impact can be interpreted as a way to measure or discover which results have been obtained using the data, how many (2011) Bioinformatics Kafkas, et al (2015) Biomedicine King (1995) Information Studies Klump, et al (2016) Earth Science Koers (2015) Clinical Epidemiology Mayernik (2012) Information Studies (Mayernik, et al, 2017) Information Studies Mooney and Newton (2012) Data Publishing Nature Physics Editorial (2016) Physics Parson (2012) Earth Science Parsons and Fox (2013) Data Publishing Peters et al (2016) Information Studies Pr€ oll and Rauber (2013) Computer Science Sieber and Trumbo (1995) Information Studies Silvello and Ferro (2016) Digital Libraries Silvello (2015) Information Retrieval Silvello (2017) Computer Science Simons, Visser, and Searle, (2013) Digital Libraries Starr et al (2015) Computer Science Thorisson (2009) Biotechnology White (1982 Social Sciences Wormack (2015) Information Studies Zw€ olf et al (2016) Spectroscopy times they have been used, where they have been used, and so on. Note that in the literature, data impact is, frequently, intended as the rate of data usage and not as a quality measure.…”

“…This is also due to the fact that data citations "are useful for the discovery of links, but less so for understanding why the link was made" . Mayernik et al (2017Mayernik et al ( , p. 1354 are among the few hypothesizing that data may be cited as "forms of reward to data providers," as a "quid pro quo for the data provider support," or as a "persuasive statement illustrating how data of high visibility or sufficient quality underlie a scientific result." Up to now, we can only find isolated and sporadic efforts to understand why data are cited.…”

Assessing the uptake of persistent identifiers by research infrastructure users

“…We use the term “research infrastructures” as an overarching grouping of multiple kinds of resources, building on terminology established in international contexts [20,21]. “Research infrastructures” include scientific equipment, collections and archives of scientific data or software, computing systems, and any other tools that enable research.…”