The Pros and Cons of the Use of Altmetrics in Research Assessment

Abstract

Many indicators derived from the web have been proposed to supplement citation-based indicators in support of research assessments. These indicators, often called altmetrics, are available commercially from Altmetric.com and Elsevier's Plum Analytics or can be collected directly. These organisations can also deliver altmetrics to support institutional self-evaluations. The potential advantages of altmetrics for research evaluation are that they may reflect important non-academic impacts and may appear before citations when an article is published, thus providing earlier impact evidence. Their disadvantages often include susceptibility to gaming, data sparsity, and difficulties translating the evidence into specific types of impact. Despite these limitations, altmetrics have been widely adopted by publishers, apparently to give authors, editors and readers insights into the level of interest in recently published articles. This article summarises evidence for and against extending the adoption of altmetrics to research evaluations. It argues that whilst systematically-gathered altmetrics are inappropriate for important formal research evaluations, they can play a role in some other contexts. They can be informative when evaluating research units that rarely produce journal articles, when seeking to identify evidence of novel types of impact during institutional or other self-evaluations, and when selected by individuals or groups to support narrative-based non-academic claims. In addition, Mendeley reader counts are uniquely valuable as early (mainly) scholarly impact indicators to replace citations when gaming is not possible and early impact evidence is needed. Organisations using alternative indicators need recruit or develop in-house expertise to ensure that they are not misused, however. Policy highlightsAltmetrics, or alternative indicators for research outputs, have been proposed as a partial solution to two research management problems: (a) assessing the societal impacts of research, and (b) obtaining early impact evidence. This article reviews the evidence and finds limited support for (a) but strong support for (b). Organisations will need to assess whether the value provided by alternative indicators in terms of helping to provide data so support research assessments is sufficient for their financial and time costs. Those using alternative indicators will deed to develop in-house expertise so that they can be used responsibly and interpreted effectively.

Do Online Readerships Offer Useful Assessment Tools? Discussion Around the Practical Applications of Mendeley Readership for Scholarly Assessment

“…Mendeley readership are considered the most prominent altmetric source with evaluative value, particularly given their large coverage of scientific publications (Costas, Zahedi, and Wouters, 2015a;Thelwall and Sud, 2016;Thelwall, 2017b), the high density levels (Mohammadi et al, 2017;Zahedi and Haustein, 2018), moderate correlation levels with citations (Zahedi, Costas, and Wouters, 2014;Costas, Zahedi, and Wouters, 2015b;Thelwall, 2018), and conceptual proximity to citation indicators (Wouters, Zahedi, and Costas, 2019;Sugimoto et al, 2017). All these interesting properties of Mendeley readership for research evaluation have been discussed in multiple scattered scientific publications (Thelwall, 2020;Thelwall, 2018), and a PhD Thesis , however we were still lacking a focused discussion on the specific possibilities of Mendeley readership for evaluative purposes. In this work we focus on illustrating the practical possibilities of Mendeley readership for research evaluation in aspects in which citation analysis pose more challenges.…”

“…In this work we focus on illustrating the practical possibilities of Mendeley readership for research evaluation in aspects in which citation analysis pose more challenges. In this regard we are adopting a relatively conservative perspective, seeing Mendeley readership as a complement to citations, which is also the most common recommendation in the literature (Haustein, Bowman, and Costas, 2016;Thelwall, 2020). There have been also discussions about the possibility of considering readership as another type of "currency of science" that could be discussed on par with citations (Costas, Perianes-Rodríguez, and Ruiz-Castillo, 2017), but we are not discussing this approach in this study.…”

“…These new indicators have gained significant attention in recent years, particularly in some national research assessment exercises (e.g., Research Excellence Framework (REF 1 ) in the UK or the Dutch Standard Evaluation Protocol (SEP 2 )), since they were expected to reflect a broader perspective on the impact of research. Mendeley readership metrics have been discussed as one of the altmetric sources with a stronger relevance for research evaluation (Thelwall, 2020), particularly as a complement in those aspects where citations present more weaknesses (Thelwall, 2017b). 3 Despite this strong interest on Mendeley readership, and discussions on their complementarity to citations, a reflection on the current best practical applications of readership is still missing in the literature.…”

A critical review on altmetrics: can we measure the social impact factor?

“…Beyond their incorporation in CVs, there are authors who propose using altmetrics as criteria for distributing funds for grants or projects [ 32 , 84 ]. They base this opinion on the altmetrics’ immediate character.…”

“…This type of detail is defined by each provider and is not explained, resulting in a lack of transparency [ 79 , 79 ]. Given the diversity of sources, collecting all the inputs takes a lot of time and this also contributes to the lack of consistency between providers [ 84 ]. Table 4 shows the altmetric data for the article “What the radiologist should know about artificial intelligence—an ESR White paper”, published in Insights into Imaging.…”