Evaluation practices and effects of indicator use—a literature review

Abstract

This review of the international literature on evaluation systems, evaluation practices and metrics (mis-)uses was written as part of a larger review commissioned by the Higher Education Funding Council for England (HEFCE) to inform their independent assessment of the role of metrics in research evaluation (2014-2015). The literature on evaluation systems, practices and effects of indicator uses is extremely heterogeneous: it comprises hundreds of sources published in different media, spread over disciplines, and with considerable variation in the nature of the evidence. A condensation of the state-of-the-art in relevant research is therefore highly timely. Our review presents the main strands in the literature, with a focus on empirical materials about possible effects of evaluation exercises, 'gaming' of indicators, and strategic responses by scientific communities and others to requirements in research assessments. In order to increase visibility and availability, an adapted and updated review is presented here as a stand-aloneafter authorisation by HEFCE.

Unique, but still best practice? The Research Excellence Framework (REF) from an international perspective

“…The supplementary literature review for The Metric Tide report , however, describes it as an evaluation system and discusses its effects in its chapter on "Strategic behaviour and goal displacement". The same literature review has since then appeared in updated version as a scholarly journal article on "Evaluation practices and effects of indicator use" (de Rijcke et al, 2016) in which it is theoretically argued that any use of metrics in a funding regime can be considered as evaluative. I will shortly describe the Norwegian model before I return to the discussion of effects, both empirically and theoretically.…”

“…The effects of the Norwegian model are also discussed in the above-mentioned article on evaluation practices and effects of indicator use (de Rijcke et al, 2016), which presents the results of the literature review for The Metric Tide report within a theoretical framework in which any use of metrics can be considered as evaluative in itself (Dahler-Larsen, 2014) by having constitutive effects on research practices, including strategic behaviour and goal displacement, task reduction and potential biases against interdisciplinary research, just to mention some of the negative effects reported in this review. I agree with this theoretical position.…”

“…Even so, Norway is given as an example along with Denmark, Flanders and Spain "that researchers' quality considerations may be displaced by incentives to produce higher quantities of publications when funding is explicitly linked to research output" (de Rijcke et al, 2016). However, none of the six publications cited to support this statement prove that it is true.…”

“…Unlike her successors, Butler could respond in her work to feedback from the participating countries. From reading the results of the evaluation of the PRFS in Norway, de Rijcke et al (2016) note that the Norwegian model is used "despite explicit advice against adoption on lower levels" (Aagaard, 2015), but are not informed about how this problem has been dealt with after the evaluation. It seems that academic literature reviews are not able, as a method, to keep trace of continuously changing national PRFS and thereby to inform mutual learning.…”

“…Hicks (2012) defines PRFS as related to both purposes; they are "national systems of research output evaluation used to distribute research funding to universities". Within a theoretical framework for regarding the two purposes as inseparable, "metrics (mis)uses" are no more than possible (mal)practices in the general domain of research evaluation systems (de Rijcke et al, 2016). I will return below to a discussion of this theoretical perspective.…”