Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data

Abstract

In this study, we address the question whether (and to what extent, respectively) altmetrics are related to the scientific quality of papers (as measured by peer assessments). Only a few studies have previously investigated the relationship between altmetrics and assessments by peers. In the first step, we analyse the underlying dimensions of measurement for traditional metrics (citation counts) and altmetrics–by using principal component analysis (PCA) and factor analysis (FA). In the second step, we test the relationship between the dimensions and quality of papers (as measured by the post-publication peer-review system of F1000Prime assessments)–using regression analysis. The results of the PCA and FA show that altmetrics operate along different dimensions, whereas Mendeley counts are related to citation counts, and tweets form a separate dimension. The results of the regression analysis indicate that citation-based metrics and readership counts are significantly more related to quality, than tweets. This result on the one hand questions the use of Twitter counts for research evaluation purposes and on the other hand indicates potential use of Mendeley reader counts.

An explorative study on a university’s outreach in the field of UN Sustainable Development Goal 2

“…Our results demonstrate that there is no clear relation between citation metrics (used here as proxy for research quality) with social media and news media mentions, two proxies of societal impact via media. Similar results have been reported using Altmetric data in Bornmann and Haunschild 8 and in Jabaley et al 9 It can therefore be assumed that research quality itself is not sufficient for societal impact, which may be dependent on factors that could not be further explored in this study, such as active outreach to news media and social media by researchers or research institutes, and media culture.…”

Does the public discuss other topics on climate change than researchers? A comparison of explorative networks based on author keywords and hashtags

“…These measures can be used as an indication of attention to the publications. One of the problems with most altmetrics is currently that it is not clear what is measured (Bornmann & Haunschild, 2018a):…”

F1000Prime recommended articles and their citations: an exploratory study of four journals

“…Contrary to their expectations, their results suggested that F1000Prime rating scores as quality proxies of the papers played less of a role in the later citation counts than journal impact factor. Also, as a quality proxy, Bornmann and Haunschild (2018) found that the factor (citation and reader impact) was strongly correlated to paper quality using the negative binomial regression between F1000Prime scores and other citation-based metrics. They acknowledged the limitation of using F1000Prime data as a proxy for paper's quality.…”