Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’

Riley, Barbara; Kernoghan, Alison; Stockton, Lisa; Montague, Steve; Yessis, Jennifer; Willis, Claire

doi:10.1093/reseval/rvx037

Cited by 32 publications

(37 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In addition, some studies used frameworks based on a logic model approach to describe the intended outputs, outcomes and impacts of a specific portfolio of research, sometimes including multiple categories of impact [44, 78, 98, 114] or focussing on policy impacts alone [99, 100] . Finally, there were several examples of studies utilising frameworks based on contribution analysis, an approach to exploring cause and effect by assessing the contribution a programme is making to observed results [ 33 ]. Such frameworks emphasise the networks and relationships associated with research production and focus on the processes and pathways that lead to impact rather than outcomes [ 27 , 33 ].…”

Section: Resultsmentioning

confidence: 99%

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Newson

King

Rychetnik

et al. 2018

Health Res Policy Sys

View full text Add to dashboard Cite

BackgroundMeasuring the policy and practice impacts of research is becoming increasingly important. Policy impacts can be measured from two directions – tracing forward from research and tracing backwards from a policy outcome. In this review, we compare these approaches and document the characteristics of studies assessing research impacts on policy and the policy utilisation of research.MethodsKeyword searches of electronic databases were conducted in December 2016. Included studies were published between 1995 and 2016 in English and reported methods and findings of studies measuring policy impacts of specified health research, or research use in relation to a specified health policy outcome, and reviews reporting methods of research impact assessment. Using an iterative data extraction process, we developed a framework to define the key elements of empirical studies (assessment reason, assessment direction, assessment starting point, unit of analysis, assessment methods, assessment endpoint and outcomes assessed) and then documented the characteristics of included empirical studies according to this framework.ResultsWe identified 144 empirical studies and 19 literature reviews. Empirical studies were derived from two parallel streams of research of equal size, which we termed ‘research impact assessments’ and ‘research use assessments’. Both streams provided insights about the influence of research on policy and utilised similar assessment methods, but approached measurement from opposite directions. Research impact assessments predominantly utilised forward tracing approaches while the converse was true for research use assessments. Within each stream, assessments focussed on narrow or broader research/policy units of analysis as the starting point for assessment, each with associated strengths and limitations. The two streams differed in terms of their relative focus on the contributions made by specific research (research impact assessments) versus research more generally (research use assessments) and the emphasis placed on research and the activities of researchers in comparison to other factors and actors as influencers of change.ConclusionsThe Framework presented in this paper provides a mechanism for comparing studies within this broad field of research enquiry. Forward and backward tracing approaches, and their different ways of ‘looking’, tell a different story of research-based policy change. Combining approaches may provide the best way forward in terms of linking outcomes to specific research, as well as providing a realistic picture of research influence.Electronic supplementary materialThe online version of this article (10.1186/s12961-018-0310-4) contains supplementary material, which is available to authorized users.

show abstract

Section: Resultsmentioning

confidence: 99%

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Newson

King

Rychetnik

et al. 2018

Health Res Policy Sys

View full text Add to dashboard Cite

show abstract

“…Reed, Bryce and MacHen, 2018), knowledge translation and utilisation (Kok and Schuit, 2012;Redman et al, 2015;Bone et al, 2017), or theories of innovation (Riley et al, 2018;Edwards and Meagher, 2019), process-oriented frameworks argue that a description of who did what, when, how and in which context, should be sufficient to assess the likeliness of future impact (CGIAR, 2008). Learning and improvement is the main motivation for opening the black box of mechanisms and factors which contribute to impact; nevertheless, identifying these "anticipatory efforts" can also be used to hold researchers or funders accountable for factors for which they are responsible (CGIAR, 2008;Kok and Schuit, 2012).…”

Section: Overview Of Existing Practicesmentioning

confidence: 99%

“…The policy process involves complex and dynamic interconnections among different activities and stakeholders: usually, research is only one piece of the wide array of elements factoring in policy decisions (Riley et al, 2018). Because research is intertwined with other factors, and because of the usually long time-lag between research and a specific impact, it is often impossible to ascertain the attribution of policy changes to specific research outputs (Boaz, Fitzpatrick and Shaw, 2009;Spaapen and van Drooge, 2011).…”

Section: Conceptual Framework: Understanding the Contribution Of Resementioning

confidence: 99%

“…Therefore, some have suggested to shift the focus of impact evaluation away from attribution to concentrate instead on contribution, thereby emphasising the processes of knowledge production and exchange leading to impact (Newson et al, 2018). Contribution approaches are interested in the "small steps" and contextual factors which are known to influence research utilisation in policy (Spaapen and van Drooge, 2011;Riley et al, 2018); they are the basis for the process-oriented evaluation frameworks introduced in the previous section.…”

Section: Conceptual Framework: Understanding the Contribution Of Resementioning

confidence: 99%

See 1 more Smart Citation

Evaluating the anticipated policy impact of research outputs: introducing the SPEIDY framework

Saidi¹

2020

Preprint

View full text Add to dashboard Cite

The Rural and Environment Science and Analytical Services (RESAS) division of the Scottish Government funds research expected to directly support the Government’s needs for science-based policy. However, the pathway from research to policy impact is rarely linear, with impact often resulting from the interrelationships of many factors. Attributing a change in policy to research activities is therefore not straightforward, especially if, like RESAS, one wants to evaluate impact in a short timeframe after a piece of research is completed. Therefore, this study aims to scope for an appropriate evaluation framework to assess the anticipated policy impact of research outputs generated by RESAS’ research providers. It intends to provide researchers with timely information on the expected contribution of their outputs to policymaking and the extent to which these contributions are likely to be realised: its main purposes are therefore learning and improvement. The framework was developed based on practices in Scottish Centres of Expertise and other organisations globally, as well as on current knowledge regarding the contribution of research to policy. Factors influencing the likelihood of research utilisation in policymaking were identified across the formulation, production and dissemination phases of research. The author is proposing a framework – termed Single-Product Evaluation for Immediate DeliverY (SPEIDY) – which assesses these factors in order to provide a picture of the likelihood of research use, and which allows to characterise this use along four categories: conceptual, instrumental, political and/or imposed. The evaluation relies on a survey collecting feedback from the policymaking team, which takes less than four minutes to complete and occurs at the time of final validation of the output. The framework, which contains both a conceptual part laying out impact-related factors and an analytical part detailing the feedback survey process, participates in current efforts to mainstream evidence-based evaluation practices. It addresses current barriers to impact evaluation in the RESAS’ research providers and its implementation could facilitate the uptake of best practices contributing to research impact.

show abstract

“…Evaluating the progress achieved thanks to policy advocacy initiatives presents numerous challenges due to: the multiple external factors that influence systems; the time frame (often requires fast adaptation to context but takes long time to produce outcomes); the myriad of actors, audiences, and interactions involved; and the fact that many tactics happen behind closed doors (Glass, ). Policy change is also a highly context‐sensitive phenomenon, which brings its own challenges (Riley et al, ). The following quote illustrates well the set of challenges faced when evaluating policy advocacy: “Evaluators must acquire and accurately weigh and synthesize imperfect information, from biaised sources with incomplete knowledge, under rapidly changing circumstances where causal links are almost impossible to establish” (Teles & Schmitt, ; Gardner & Brindis, , p. 76).…”

Section: Introductionmentioning

confidence: 99%

Contribution of the Alive & Thrive–UNICEF advocacy efforts to improve infant and young child feeding policies in Southeast Asia

Michaud-Létourneau

Gayard

Pelletier

2019

Maternal & Child Nutrition

View full text Add to dashboard Cite

Evaluating the impact of advocacy for policy change presents many challenges. Recent advances in the field of evaluation, such as contribution analysis (CA), offer guidance on how to make credible claims regarding such impact. The purposes of this article are (a) to detail the application of CA to assess the contribution of an advocacy initiative to improve infant and young child feeding policies and (b) to present the emergent theory of change and contribution story of how progress was achieved. An evaluation applying developmental evaluation and CA was conducted on the Alive & Thrive (A&T)–UNICEF initiative in seven Southeast Asian countries to document the extent to which policy objectives were achieved and identify key drivers of policy change. A contribution story was developed based on these experiences. The advocacy approach, which involved a four‐part process, contributed directly to (a) set the agenda of various actors and (b) create a strategic group; and indirectly to (a) set and maintain the issue on the agenda at all stages of the policy cycle, (b) support the government to carry out a set of critical tasks, and (c) extend commitment. All of this helped to achieve progress towards policy change. External influences were at play. The flexibility of A&T allowed key actors to utilize the positive external influences and address some of the negative ones through developing responsive strategies mitigating their effects. The emerging contribution story supports that A&T–UNICEF initiative contributed to the progress achieved in the participating countries.

show abstract

Using contribution analysis to evaluate the impacts of research on policy: Getting to ‘good enough’

Cited by 32 publications

References 22 publications

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research

Evaluating the anticipated policy impact of research outputs: introducing the SPEIDY framework

Contribution of the Alive & Thrive–UNICEF advocacy efforts to improve infant and young child feeding policies in Southeast Asia

Contact Info

Product

Resources

About