Developing a methodology to assess the impact of research grant funding: A mixed methods approach

Bloch, Carter Walter; Sørensen, Mads Peter; Graversen, Ebbe Krogh; Schneider, Jesper W.; Schmidt, Evanthia Kalpazidou; Aagaard, Kaare; Mejlgaard, Niels

doi:10.1016/j.evalprogplan.2013.12.005

Cited by 49 publications

(42 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Formative evaluations findings can guide the effective replication of programmes in other settings [7]. Such formative evaluations are often designed as multiple, mixed-methods evaluations [8], defined as the application of a combination of qualitative and quantitative approaches [9–11], to allow them to address adequately a wide range of questions regarding how a programme was implemented.…”

Section: Introductionmentioning

confidence: 99%

Multiple and mixed methods in formative evaluation: Is more better? Reflections from a South African study

Odendaal

Atkins

Lewin

2016

BMC Med Res Methodol

View full text Add to dashboard Cite

BackgroundFormative programme evaluations assess intervention implementation processes, and are seen widely as a way of unlocking the ‘black box’ of any programme in order to explore and understand why a programme functions as it does. However, few critical assessments of the methods used in such evaluations are available, and there are especially few that reflect on how well the evaluation achieved its objectives. This paper describes a formative evaluation of a community-based lay health worker programme for TB and HIV/AIDS clients across three low-income communities in South Africa. It assesses each of the methods used in relation to the evaluation objectives, and offers suggestions on ways of optimising the use of multiple, mixed-methods within formative evaluations of complex health system interventions.MethodsThe evaluation’s qualitative methods comprised interviews, focus groups, observations and diary keeping. Quantitative methods included a time-and-motion study of the lay health workers’ scope of practice and a client survey. The authors conceptualised and conducted the evaluation, and through iterative discussions, assessed the methods used and their results.ResultsOverall, the evaluation highlighted programme issues and insights beyond the reach of traditional single methods evaluations. The strengths of the multiple, mixed-methods in this evaluation included a detailed description and nuanced understanding of the programme and its implementation, and triangulation of the perspectives and experiences of clients, lay health workers, and programme managers. However, the use of multiple methods needs to be carefully planned and implemented as this approach can overstretch the logistic and analytic resources of an evaluation.ConclusionsFor complex interventions, formative evaluation designs including multiple qualitative and quantitative methods hold distinct advantages over single method evaluations. However, their value is not in the number of methods used, but in how each method matches the evaluation questions and the scientific integrity with which the methods are selected and implemented.

show abstract

Section: Introductionmentioning

confidence: 99%

Multiple and mixed methods in formative evaluation: Is more better? Reflections from a South African study

Odendaal

Atkins

Lewin

2016

BMC Med Res Methodol

View full text Add to dashboard Cite

show abstract

“…Governments hope that concentrating resources in the best research performers will make the science system more effective and yield more and better outcomes for the entire system. From this perspective, excellence policies are designed to produce outcomes that would not have happened without the intervention, namely system-wide additionality (Tyler et al 2009) including for example (Bloch et al 2014): a) new research activities that would not otherwise have been possible ('input additionality'); ) new products and/or publications ('output additionality'); c) research and publications strategies, international collaborations, etc. ('behavioural additionality'); d) changes in workplace mobility ('career additionality'); e) changes in institutional research environments ('institutional additionality').…”

Section: Theoretical Framework: Additionality Through Coordinationmentioning

confidence: 99%

“…This efficiency argument is common to all public funding schemes: through differentiation excellence policies create system-wide additionality, which involves outcomes that would not have happened without the policy intervention (Tyler et al, 2009;Bloch et al 2014; Bloch & Sørensen 2015).…”

Section: Introductionmentioning

confidence: 99%

Different recipes for the same dish: Comparing policies for scientific excellence across different countries

Cremonini

Horlings

Hessels

2017

Science and Public Policy

View full text Add to dashboard Cite

Abstract:Many countries witness the rise of 'excellence initiatives'. These policies promote vertical differentiation in the science system by funding top research performers and expecting positive spill-over effects. However, current understanding of the functioning and (potential) effects of these instruments is limited. We compare policies aimed at promoting excellence in four countries (the UK, Germany, Denmark and Switzerland), using secondary sources and 14 expert interviews. Using the notion of coordination approaches as a heuristic tool, we characterise each policy in terms of the coordinating actor, the system addressed, the activities that are coordinated, the specific interventions taken and the types of relationships affected. We find that countries adopt very different approaches to reach similar goals and thus bring into question appealing but simplistic ideas of 'excellence' as an agreed concept. Remarkably, excellence policies are more prone to reveal existing but tacit diversity in the system than to generate new relational patterns.http://mc.manuscriptcentral.com/spp AbstractMany countries witness the rise of 'excellence initiatives'. These policies promote vertical differentiation in the science system by funding top research performers and expecting positive spill-over effects. However, current understanding of the functioning and (potential) effects of these instruments is limited. We compare policies aimed at promoting excellence in four countries (the UK, Germany, Denmark and Switzerland), using secondary sources and 14 expert interviews. Using the notion of coordination approaches as a heuristic tool, we characterise each policy in terms of the coordinating actor, the system addressed, the activities that are coordinated, the specific interventions taken and the types of relationships affected. We find that countries adopt very different approaches to reach similar goals and thus bring into question appealing but simplistic ideas of 'excellence' as an agreed concept. Remarkably, excellence policies are more prone to reveal existing but tacit diversity in the system than to generate new relational patterns.Keywords: excellence; science policy; coordination; differentiation; Matthew Effect IntroductionIn many countries promoting scientific excellence has become the primary target of science policy. A generic measure is to make research funding conditional upon performance. Several countries have introduced performance-based research funding system in which resources are directed towards excellent performers (Hicks 2012;Lewis & Ross 2011;Lewis 2015;Jonkers & Zacharewicz 2016). In addition, funding has shifted from institutional core funding to competitive project funding (Lepori et al. 2007;Jongbloed et al 2015). In a recent report, the OECD observed the emergence of research excellence initiatives (REIs) aimed at encouraging 'outstanding research by providing large-scale, long-term funding to designated research units, with an emphasis on research of exceptional quality.' (OECD 2014, p. 2...

show abstract

“…Impact assessment involves a number of complex issues that are difficult to fully address in a single study (Bloch et al, 2014). Furthermore, conducting cost-benefit analysis and productivity analysis for research is costly (Pedersen et al, 2011).…”

Section: Impact Assessmentmentioning

confidence: 99%

“…Furthermore, conducting cost-benefit analysis and productivity analysis for research is costly (Pedersen et al, 2011). Due to this, it was decided to carry out the analysis as a mixed methods approach, including both quantitative and qualitative methods (Bloch et al, 2014).…”

Section: Impact Assessmentmentioning

confidence: 99%

Organic Research and Development in Denmark (1996-2010) – Effects on the Organic Sector and Society

Andreasen¹,

Rasmussen²,

Halberg³

2015

SAR

View full text Add to dashboard Cite

The ICROFS (International Centre for Research in Organic Food Systems) has conducted an analysis of the effects of organic research in Denmark (1996Denmark ( -2010 on the Danish organic sector and on society in general. Over these 15 years, three national programs and one program with European collaboration have been implemented in Denmark, financed via special government grants that amounted to just over 500 million DKK (approx. € 67 million-or approximately $ 80 million). The analysis itself was carried out as a compilation of information from three perspectives, each of which has been independently documented:  Interviews with (representatives of) end-users of results from research and development (R&D) investigating their assessment of the challenges in the sector and solutions developed from 1996-2010  Assessment of the R&D endeavours in different thematic areas (dairy/milk, pigs, crops, etc.) as they related to end-users and the stated challenges at that time  Documentation of the dissemination of R&D results in relation to themes and challenges in the sectorThe results showed very good correspondence between end-users' perceptions of the challenges overcome in the sector, the R&D initiated in the research programmes, and the dissemination of research results and other forms of knowledge transfer. The analysis documented direct effects of the research initiatives targetting the challenges in the sector such as higher yields, weed and pest control, animal health and welfare, the potential for phasing out the use of antibiotics in Danish dairy herds and reducing the problems caused by seedborne diseases. It also describes where research did not contribute as much to overcoming challenges. In contrast, the analysis showed that the effects of the research in the organic processing industry and among relevant governmental and non-governmental organisations were of a more indirect character. Research has helped stabilize the supply and quality of raw materials at a time of growing demand and sales. Organic research also generates new knowledge and leads to new opportunities that can provide inspiration for a green conversion, product diversification and growth also in conventional agriculture. The analysis showed that research under the national research programs overall have been very applied and directed at the barriers in the sector in order to support the general market and growth conditions for the organic sector. Having laid a solid foundation, the private sector has been able to take advantage of commercial opportunities when demand grew, while adhering to the organic policy objectives of the market-driven growth in the organic sector.

show abstract

Developing a methodology to assess the impact of research grant funding: A mixed methods approach

Cited by 49 publications

References 32 publications

Multiple and mixed methods in formative evaluation: Is more better? Reflections from a South African study

Multiple and mixed methods in formative evaluation: Is more better? Reflections from a South African study

Different recipes for the same dish: Comparing policies for scientific excellence across different countries

Organic Research and Development in Denmark (1996-2010) – Effects on the Organic Sector and Society

Contact Info

Product

Resources

About