Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Abstract

BackgroundMeasuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research.MethodsKeyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework.ResultsWe identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change.ConclusionsThe Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence.Electronic supplementary materialThe online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.

Recent Trends and Future Direction of Dental Research in the Digital Era

“…Scientists in dental medicine have also to be aware that funding might be increasingly dependent on the possibility to demonstrate an impact on a large scale. Thus, the use of impact assessments in the future will most likely serve the following two tasks: (1) demonstrating the value of research, and (2) increasing the value of research through a more effective way of financing research in order to have a societal impact [48,49].…”

Mapping the Qualitative Evidence Base on the Use of Research Evidence in Health Policy-Making: A Systematic Review

“…We also searched the reference lists of all included studies and of previous reviews whose subject matter focus had similarities with the present review. [16][17][18][19][20][21]47,49 Experts and colleagues were contacted to obtain information about any as yet unidentified studies. Furthermore, we screened an inventory of studies of evidence-to-policy processes of which we were already aware.…”

Evaluating the anticipated policy impact of research outputs: introducing the SPEIDY framework

“…Because research is intertwined with other factors, and because of the usually long time-lag between research and a specific impact, it is often impossible to ascertain the attribution of policy changes to specific research outputs (Boaz, Fitzpatrick and Shaw, 2009;Spaapen and van Drooge, 2011). Therefore, some have suggested to shift the focus of impact evaluation away from attribution to concentrate instead on contribution, thereby emphasising the processes of knowledge production and exchange leading to impact (Newson et al, 2018). Contribution approaches are interested in the "small steps" and contextual factors which are known to influence research utilisation in policy (Spaapen and van Drooge, 2011;Riley et al, 2018); they are the basis for the process-oriented evaluation frameworks introduced in the previous section.…”