The future of societal impact assessment using peer review: pre-evaluation training, consensus building and inter-reviewer reliability

Abstract

There are strong political reasons underpinning the desire to achieve a high level of inter-reviewer reliability (IRR) within peer review panels. Achieving a high level of IRR is synonymous with an efficient review system, and the wider perception of a fair evaluation process. Therefore, there is an arguable role for a more structured approach to the peer review process during a time when evaluators are effectively novices in practice with the criterion, such as with societal impact. This article explores the consequences of a structured peer review process that aimed to increase inter-reviewer reliability within panels charged with assessing societal impact. Using a series of interviews from evaluators from the UK's Research Excellence Framework conducted before (pre-evaluation) and then again after the completion of the process (post-evaluation), it explores evaluators' perceptions about how one tool of a structured evaluation process, pre-evaluation training, influenced their approaches to achieving a consensus within the peer review panel. Building on lessons learnt from studies on achieving inter-reviewer reliability and from consensus building with peer review groups, this article debates the benefits of structured peer review processes in cases when the evaluators are unsure of the criterion (as was the case with the Impact criterion), and therefore the risks of a low IRR are increased. In particular, this article explores how individual approaches to assessing Impact were normalized during group deliberation around Impact and how these relate to evaluators' perceptions of the advice given during the pre-evaluation training. This article is published as part of a collection on the future of research assessment.

The Corona-Eye: Exploring the risks of COVID-19 on fair assessments of impact for REF2021

“…The use of calibration exercises prior to the formal evaluation taking place within panels (Derrick and Samuel 2017), was used successfully during REF2014 as a mechanism to assist robust discussion around Impact, as well as provide an opportunity for panel members to clarify expectations and form a common lens to guide the impact evaluation (Derrick 2018). Calibration exercises, especially when the evaluation is anticipated as more complex, as was the case of Impact in REF2014, are used as an exercise in maintaining consistency and fairness in evaluation.…”

The dilemmas and uncertainties in assessing the societal impact of research

“…Moreover, in practice, case studies often utilize academic bibliometrics instead of social metrics, making it more difficult to understand the verification logic of impact. The pursuit of normalizing impact criteria and evaluation mechanics also encounter academic peer-review, which is often used as expertise in case studies and other forms of panel assessment (Derrick & Samuel 2017;Derrick 2018, 11).…”

“…It seems that interdisciplinarity as a form of academic self-control is easier to comprehend than external social control indicating lack of assessment culture for such conceptualization (cf. Derrick & Samuel 2017). Ostensibly, academic differentiation seems not to fit in the frame of impact expertise emphasizing multidisciplinary aims of knowledge, but, yet, disciplinary knowledge and understanding had their place in the informants' views.…”

Editorial

“…To have a foundation guide for evaluators, the alignment of what should be considered impact is necessary to reach a consensus in the evaluation, regardless of their diverse opinions and ideas. Derrick and Samuel (2017) suggest, for example, having pre-evaluation training.…”