Evaluating the anticipated policy impact of research outputs: introducing the SPEIDY framework

Abstract

The Rural and Environment Science and Analytical Services (RESAS) division of the Scottish Government funds research expected to directly support the Government’s needs for science-based policy. However, the pathway from research to policy impact is rarely linear, with impact often resulting from the interrelationships of many factors. Attributing a change in policy to research activities is therefore not straightforward, especially if, like RESAS, one wants to evaluate impact in a short timeframe after a piece of research is completed. Therefore, this study aims to scope for an appropriate evaluation framework to assess the anticipated policy impact of research outputs generated by RESAS’ research providers. It intends to provide researchers with timely information on the expected contribution of their outputs to policymaking and the extent to which these contributions are likely to be realised: its main purposes are therefore learning and improvement. The framework was developed based on practices in Scottish Centres of Expertise and other organisations globally, as well as on current knowledge regarding the contribution of research to policy. Factors influencing the likelihood of research utilisation in policymaking were identified across the formulation, production and dissemination phases of research. The author is proposing a framework – termed Single-Product Evaluation for Immediate DeliverY (SPEIDY) – which assesses these factors in order to provide a picture of the likelihood of research use, and which allows to characterise this use along four categories: conceptual, instrumental, political and/or imposed. The evaluation relies on a survey collecting feedback from the policymaking team, which takes less than four minutes to complete and occurs at the time of final validation of the output. The framework, which contains both a conceptual part laying out impact-related factors and an analytical part detailing the feedback survey process, participates in current efforts to mainstream evidence-based evaluation practices. It addresses current barriers to impact evaluation in the RESAS’ research providers and its implementation could facilitate the uptake of best practices contributing to research impact.