Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements

Abstract

How does scientific research affect the world around us? Being able to answer this question is of great importance in order to appropriately channel efforts and resources in science. The impact by scientists in academia is currently measured by citation based metrics such as h-index, i-index and citation counts. These academic metrics aim to represent the dissemination of knowledge among scientists rather than the impact of the research on the wider world. In this work we are interested in measuring scientific impact beyond academia, on the economy, society, health and legislation (comprehensive impact). Indeed scientists are asked to demonstrate evidence of such comprehensive impact by authoring case studies in the context of the Research Excellence Framework (REF). We first investigate the extent to which existing citation based metrics can be indicative of comprehensive impact. We have collected all recent REF impact case studies from 2014 and we have linked these to papers in citation networks that we constructed and derived from CiteSeerX, arXiv and PubMed Central using a number of text processing and information retrieval techniques. We have demonstrated that existing citation-based metrics for impact measurement do not correlate well with REF impact results. We also consider metrics of online attention surrounding scientific works, such as those provided by the Altmetric API. We argue that in order to be able to evaluate wider non-academic impact we need to mine information from a much wider set of resources, including social media posts, press releases, news articles and political debates stemming from academic work. We also provide our data as a free and reusable collection for further analysis, including the PubMed citation network and the correspondence between REF case studies, grant applications and the academic literature.

Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)

“…Large-scale, systematic studies of altmetrics are necessary to reveal evidence of their validity as an impact indicator (Haustein, Peters, Sugimoto, Thelwall, & Larivière, 2014). In this study, (following earlier studies by Bornmann, 2014bBornmann, , 2015bRavenscroft et al, 2017), we address the question of convergent validity of altmetrics data for indexing societal impact by:…”

“…In the UKREF, societal impact assessments are now a budget-relevant part of evaluation (NISO Alternative Assessment Metrics Project, 2014). By implication, impact assessment in the REF (and other similar national evaluation procedures) is about a social contract: "a relationship through which researchers gain funding in return for contributing to society"(Terama et al, 2017) Ravenscroft et al (2017). investigated whether altmetrics might replace reviewers' assessments of societal impact by focusing on references cited in case studies.…”

Systematic analysis of agreement between metrics and peer review in the UK REF

“…Finally, as a limitation of our work, we emphasise that we do not consider the broader societal, cultural, and economic impact that is also evaluated in the REF. Such a broader evaluation cannot be done on the basis of metrics (Ravenscroft et al, 2017;Bornmann, Haunschild, and Adams, 2018;Pollitt et al, 2016) and should therefore be carried out using peer review. Outputs that are not covered in bibliographic databases such as the WoS, Scopus, Dimensions, and Microsoft Academic also need to be assessed by peer review.…”

Go Wide, Go Deep: Quantifying the Impact of Scientific Papers Through Influence Dispersion Trees

“…Chakraborty et al [5] showed that the change in yearly citation count of articles published in journals is different from articles published in conferences. Even the evolution of yearly citation count of papers varies across disciplines [6,34]. This further raises a new proposition of designing domain-specific impact measurement metrics.…”