Assessing the Impact and Quality of Research Data Using Altmetrics and Other Indicators

Abstract

Research data in all its diversity-instrument readouts, observations, images, texts, video and audio files, and so on-is the basis for most advancement in the sciences. Yet the assessment of most research programmes happens at the publication level, and data has yet to be treated like a first-class research object. How can and should the research community use indicators to understand the quality and many potential impacts of research data? In this article, we discuss the research into research data metrics, these metrics' strengths and limitations with regard to formal evaluation practices, and the possible meanings of such indicators. We acknowledge the dearth of guidance for using altmetrics and other indicators when assessing the impact and quality of research data, and suggest heuristics for policymakers and evaluators interested in doing so, in the absence of formal governmental or disciplinary policies. Policy highlights • Research data is an important building block of scientific production, but efforts to develop a framework for assessing data's impacts have had limited success to date. • Indicators like citations, altmetrics, usage statistics, and reuse metrics highlight the influence of research data upon other researchers and the public, to varying degrees. • In the absence of a shared definition of "quality", varying metrics may be used to measure a dataset's accuracy, currency, completeness, and consistency. • Policymakers interested in setting standards for assessing research data using indicators should take into account indicator availability and disciplinary variations in the data when creating guidelines for explaining and interpreting research data's impact. • Quality metrics are context dependent: they may vary based upon discipline, data structure, and repository. For this reason, there is no agreed upon set of indicators that can be used to measure quality. • Citations are well-suited to showcase research impact and are the most widely understood indicator. However, efforts to standardize and promote data citation practices have seen limited success, leading to varying rates of citation data availability across disciplines. • Altmetrics can help illustrate public interest in research, but availability of altmetrics for research data is very limited. • Usage statistics are typically understood to showcase interest in research data, but infrastructure to standardize these measures have only recently been introduced, and not all repositories report their usage metrics to centralized data brokers like DataCite. • Reuse metrics vary widely in terms of what kinds of reuse they measure (e.g. educational, scholarly, etc). This category of indicator has the fewest heuristics for collection and use associated with it; think about explaining and interpreting reuse with qualitative data, wherever possible. • All research data impact indicators should be interpreted in line with the Leiden Manifesto's principles, including accounting for disciplinary variation and data availability. • Assessing research...

The data‐index: An author‐level metric that values impactful data and incentivizes data sharing

“…Our call to better reward authors for producing and sharing impactful data echoes the calls of others before us, including those that developed the FAIR (Findable, Accessible, Interoperable, Reusable) Guiding Principles for data management (Ewers et al., 2019; Konkiel, 2020; Wilkinson et al., 2016). In the last decade, huge efforts have been made to increase the recognition of datasets, including the development of dataset repositories, the largest of which are DataCite and Thomson Reuters Data Citation Index (Arend et al., 2020; Cousijn et al., 2019; Konkiel, 2020; Pavlech, 2016). These repositories, respectively, store 21.8 and 10.3 million datasets as first‐class research outputs (“Data Citation Index,” 2021, “DataCite,” 2021).…”

“…A final consideration is what the unit of a dataset is, and defining this is not as straightforward as defining a publication (Konkiel, 2020). It raises questions such as.…”

“…persistent identifiers such as DOIs), citations of these datasets are tracked, and guidelines for citing data have been developed (Cousijn et al., 2018, 2019). Citations are well‐suited to showcase research impact as they are the most widely understood indicator for data, and these repositories have greatly increased the recognition of datasets (Konkiel, 2020). However, researchers are yet to be meaningfully recognized for their contributions; two recent workshops with 70 and 32 stakeholders recommended prioritizing developing metrics as a way to give credit to researchers for generating and sharing data (Federer, 2020; Pierce et al., 2019).…”

Trustworthy or not? Research data on COVID-19 in data repositories

“…These dimensions are not specific for research data. What is specific and special with research data has been described as their “contextual quality” ( Stausberg et al, 2019 , following Wang and Strong, 1996 ), because of the particular and often dynamic nature of datasets in given discipline environments, communities, and infrastructures; especially in research fields that produce unstructured and semistructured data, manual data quality checks are considered an important safeguard against fraud ( Konkiel, 2020 ).…”

Istraživački podatci hrvatskih autora na platformi Web of Science

“…Po tome se zapisi o istraživačkim podatcima razlikuju od onih za članke ili monografije koji su po naravi fiksni. 34 To tzv. verzioniranje istraživačkih podataka, odnosno proces stvaranja ili upravljanja promjenama datoteke tijekom istraživanja ili projekta 13 može se ilustrirati primjerom repozitorija Zenodo kojim se, prema našim rezultatima, koristi najveći broj hrvatskih znanstvenika (tablica 2).…”