Measuring the success of facilitated engagement between knowledge producers and users: a validated scale

Dwan, Kathryn; McInnes, Peter; Mazumdar, Soumya

doi:10.1332/174426414x14165029835102

Cited by 11 publications

(12 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… Brennan et al, 2015 [ 110 ] 1, 6, 16 ✓ ✓ ✓ ✓ 17. Dwan et al, 2015 [ 128 ] 4, 15 ✓ ✓ ✓ 18. Shroff et al, 2015 [ 103 ] 2, 6, 14, 15 ✓ ✓ ✓ ✓ ✓ ✓ 19.…”

Section: Resultsmentioning

confidence: 99%

What can we learn from interventions that aim to increase policy-makers’ capacity to use research? A realist scoping review

Haynes

Rowbotham

Redman

et al. 2018

Health Res Policy Sys

View full text Add to dashboard Cite

BackgroundHealth policy-making can benefit from more effective use of research. In many policy settings there is scope to increase capacity for using research individually and organisationally, but little is known about what strategies work best in which circumstances. This review addresses the question: What causal mechanisms can best explain the observed outcomes of interventions that aim to increase policy-makers’ capacity to use research in their work?MethodsArticles were identified from three available reviews and two databases (PAIS and WoS; 1999–2016). Using a realist approach, articles were reviewed for information about contexts, outcomes (including process effects) and possible causal mechanisms. Strategy + Context + Mechanism = Outcomes (SCMO) configurations were developed, drawing on theory and findings from other studies to develop tentative hypotheses that might be applicable across a range of intervention sites.ResultsWe found 22 studies that spanned 18 countries. There were two dominant design strategies (needs-based tailoring and multi-component design) and 18 intervention strategies targeting four domains of capacity, namely access to research, skills improvement, systems improvement and interaction. Many potential mechanisms were identified as well as some enduring contextual characteristics that all interventions should consider. The evidence was variable, but the SCMO analysis suggested that tailored interactive workshops supported by goal-focused mentoring, and genuine collaboration, seem particularly promising. Systems supports and platforms for cross-sector collaboration are likely to play crucial roles. Gaps in the literature are discussed.ConclusionThis exploratory review tentatively posits causal mechanisms that might explain how intervention strategies work in different contexts to build capacity for using research in policy-making.Electronic supplementary materialThe online version of this article (10.1186/s12961-018-0277-1) contains supplementary material, which is available to authorized users.

show abstract

“… Brennan et al, 2015 [ 110 ] 1, 6, 16 ✓ ✓ ✓ ✓ 17. Dwan et al, 2015 [ 128 ] 4, 15 ✓ ✓ ✓ 18. Shroff et al, 2015 [ 103 ] 2, 6, 14, 15 ✓ ✓ ✓ ✓ ✓ ✓ 19.…”

Section: Resultsmentioning

confidence: 99%

What can we learn from interventions that aim to increase policy-makers’ capacity to use research? A realist scoping review

Haynes

Rowbotham

Redman

et al. 2018

Health Res Policy Sys

View full text Add to dashboard Cite

show abstract

“…One‐off workshops are not generally effective (Miller et al , 2006), neither are mailings or short courses according to a review of 55 studies in the existing literature (Davis & Taylor‐Vaisey, 1997). No difference was found between the use of seminars and roundtables in encouraging the use of systematic review results in health policy (Dwan et al , 2015).…”

Section: How Effective Is Active Engagement In Deliverymentioning

confidence: 99%

What is the evidence on the best way to get evidence into use in education?

Gorard

See

Siddiqui

2020

Review of Education

View full text Add to dashboard Cite

For decades there have been calls by concerned stakeholders to improve the quality of education research, and some progress has been made towards creating a more secure evidence base in some areas. More programmes and approaches that have a reasonable evidence base are now also being used in schools (but not in policy, and not necessarily because they have a reasonable evidence base). However, there has been no equivalent improvement in secure knowledge about how best to get that evidence into use, or even what difference it makes when such evidence is used. This paper looks at what little is already known about the different ways to get research evidence into use in education by summarising the results of a large-scale review of the literature. A total of 323 of the most relevant studies were looked at across all areas of public policy, and judged for quality and contribution. Very few (33) were of the appropriate design and quality needed to make robust causal claims about evidence-intouse, and even fewer of these concerned education. This means that despite over 20 years of modest improvement in research on what works in education policy and practice, the evidence on how best to deploy these findings is still very weak. We consider studies in terms of several issues, including whether they look at changes in user knowledge and behaviour, or student outcomes, and how evidence is best modified before use. Providing access to raw research evidence or even slightly simplified evidence is not generally an effective way of getting it used, even if that evidence is presented to users by knowledge-brokers, in short courses or similar. What is more likely to work for both policy and practice is the engineering of high quality evidence into a more usable format and presenting it actively or iteratively via a respected and trusted conduit, or through population measures such as legislation. Having the users actually do the research is another promising approach. Expecting each individual study they fund to have an impact is not the way forward, as this may encourage widespread use of ineffective or even harmful interventions. Publicly-funded users, including policy-makers, should be required to use evidence-led programmes from those libraries providing them and which are appropriate and relevant to their aims. Research funders should support these approaches, and help to build up libraries of successfully tested programmes. Researchers need to be scrupulous, looking at their new evidence in the context of what is already known and not looking to obtain 'impact' from single studies. More and better research is needed on the best routes for evidence-into-use. However, the improvements required of all parties are as much ethical in nature as they are technical or scientific.Keywords use of evidence, engineering of evidence, translation of evidence, research impact, knowledge transfer, robust evidence.

show abstract

“…Widely used capacity building strategies include training for policymakers in finding and interpreting research [4–6], the provision of research resources such as databases and evidence-briefs tailored to policy needs [7–10], and partnerships between policymakers and researchers to co-produce research [11–14]. Investment in these strategies stems from a commitment in both policy and research sectors to capitalise on untapped potential for research to inform policy [15].…”

Section: Introductionmentioning

confidence: 99%

Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers’ capacity to engage with and use research

Brennan

McKenzie

Turner

et al. 2017

Health Res Policy Sys

107

View full text Add to dashboard Cite

BackgroundCapacity building strategies are widely used to increase the use of research in policy development. However, a lack of well-validated measures for policy contexts has hampered efforts to identify priorities for capacity building and to evaluate the impact of strategies. We aimed to address this gap by developing SEER (Seeking, Engaging with and Evaluating Research), a self-report measure of individual policymakers’ capacity to engage with and use research.MethodsWe used the SPIRIT Action Framework to identify pertinent domains and guide development of items for measuring each domain. Scales covered (1) individual capacity to use research (confidence in using research, value placed on research, individual perceptions of the value their organisation places on research, supporting tools and systems), (2) actions taken to engage with research and researchers, and (3) use of research to inform policy (extent and type of research use). A sample of policymakers engaged in health policy development provided data to examine scale reliability (internal consistency, test-retest) and validity (relation to measures of similar concepts, relation to a measure of intention to use research, internal structure of the individual capacity scales).ResultsResponse rates were 55% (150/272 people, 12 agencies) for the validity and internal consistency analyses, and 54% (57/105 people, 9 agencies) for test-retest reliability. The individual capacity scales demonstrated adequate internal consistency reliability (alpha coefficients > 0.7, all four scales) and test-retest reliability (intra-class correlation coefficients > 0.7 for three scales and 0.59 for fourth scale). Scores on individual capacity scales converged as predicted with measures of similar concepts (moderate correlations of > 0.4), and confirmatory factor analysis provided evidence that the scales measured related but distinct concepts. Items in each of these four scales related as predicted to concepts in the measurement model derived from the SPIRIT Action Framework. Evidence about the reliability and validity of the research engagement actions and research use scales was equivocal.ConclusionsInitial testing of SEER suggests that the four individual capacity scales may be used in policy settings to examine current capacity and identify areas for capacity building. The relation between capacity, research engagement actions and research use requires further investigation.Electronic supplementary materialThe online version of this article (doi:10.1186/s12961-016-0162-8) contains supplementary material, which is available to authorized users.

show abstract

Measuring the success of facilitated engagement between knowledge producers and users: a validated scale

Cited by 11 publications

References 33 publications

What can we learn from interventions that aim to increase policy-makers’ capacity to use research? A realist scoping review

What can we learn from interventions that aim to increase policy-makers’ capacity to use research? A realist scoping review

What is the evidence on the best way to get evidence into use in education?

Development and validation of SEER (Seeking, Engaging with and Evaluating Research): a measure of policymakers’ capacity to engage with and use research

Contact Info

Product

Resources

About