The role of metrics in peer assessments

Abstract

Metrics on scientific publications and their citations are easily accessible and are often referred to in assessments of research and researchers. This paper addresses whether metrics are considered a legitimate and integral part of such assessments. Based on an extensive questionnaire survey in three countries, the opinions of researchers are analysed. We provide comparisons across academic fields (cardiology, economics, and physics) and contexts for assessing research (identifying the best research in their field, assessing grant proposals and assessing candidates for positions). A minority of the researchers responding to the survey reported that metrics were reasons for considering something to be the best research. Still, a large majority in all the studied fields indicated that metrics were important or partly important in their review of grant proposals and assessments of candidates for academic positions. In these contexts, the citation impact of the publications and, particularly, the number of publications were emphasized. These findings hold across all fields analysed, still the economists relied more on productivity measures than the cardiologists and the physicists. Moreover, reviewers with high scores on bibliometric indicators seemed more frequently (than other reviewers) to adhere to metrics in their assessments. Hence, when planning and using peer review, one should be aware that reviewers—in particular reviewers who score high on metrics—find metrics to be a good proxy for the future success of projects and candidates, and rely on metrics in their evaluation procedures despite the concerns in scientific communities on the use and misuse of publication metrics.

Do peers share the same criteria for assessing grant applications?

“…These field differences are generally consistent with the literature, which finds that evaluation criteria differ between disciplines (e.g. Guetzkow, Lamont and Mallard, 2004;Hamann and Beljean, 2017;Hammarfelt, 2020;Hug et al, 2013;Langfeldt, Reymert and Aksnes, 2020;Reymert, Jungblut and Borlaug, 2020). The field differences between the consensus-close classes would offer a convenient explanation for the difference between the core and broad consensus, as the rating patterns of the consensus-close classes are almost identical to those of the consensus classes.…”

“…Specifically, this means that criteria related to knowledge gaps, feasibility, rigour, comprehensibility and argumentation, academic relevance, as well as to the competence and experience of the applicant should have a stronger influence than other criteria, such as originality and other contribution-related criteria. This is consistent with the notion of conservatism, which holds that innovative and ground-breaking research is undervalued in peer review (Lee et al, 2013), while other aspects, such as methodology (Lee, 2015) and feasibility (Langfeldt et al, 2020), are overweighted.…”

“…This is in stark contrast to research policy, where the notion of societal impact has gained significant traction in recent years, also with regard to the humanities (Pedersen, Grønvad and Hvidtfeldt, 2020). As funding agencies mediate between the research domain and the policy domain (Guston, 2000;Langfeldt et al, 2020;Reinhart, 2012), serious tensions regarding societal impact are to be expected in grant peer review if societal impact will be further emphasized. Humanities scholars and agencies may thus explore approaches to mitigate or avoid these looming tensions.…”

Toward More Inclusive Metrics and Open Science to Measure Research Assessment in Earth and Natural Sciences

“…These metrics form the basis of criteria that have been widely used to measure institutional reputation, as well as that of authors and research groups. By relying mostly on citations (Langfeldt et al, 2021 ), however, they are inherently flawed in that they provide only a limited picture of scholarly production. Indeed, citations only count document use within scholarly works and thus provide a very limited view of the use and impact of an article.…”

Towards more inclusive metrics and open science to measure research assessment in Earth and natural sciences

“…Submitted to SI on "Quality and Quantity in Research Assessment: Examining the Merits of Metrics" in Frontiers in Research Metrics and Analytics Those metrics and analyses are based around citations (Langfeldt et al, 2021), but citations only count document use within scholarly works and thus provide a very limited view of the use and impact of an article. Those reveal only the superficial dimensions of a research's impact on society.…”