Tackling the tensions in evaluating capacity strengthening for health research in low- and middle-income countries

Abstract

Strengthening research capacity in low- and middle-income countries is one of the most effective ways of advancing their health and development but the complexity and heterogeneity of health research capacity strengthening (RCS) initiatives means it is difficult to evaluate their effectiveness. Our study aimed to enhance understanding about these difficulties and to make recommendations about how to make health RCS evaluations more effective. Through discussions and surveys of health RCS funders, including the ESSENCE on Health Research initiative, we identified themes that were important to health RCS funders and used these to guide a systematic analysis of their evaluation reports. Eighteen reports, produced between 2000 and 2013, representing 12 evaluations, were purposefully selected from 54 reports provided by the funders to provide maximum variety. Text from the reports was extracted independently by two authors against a pre-designed framework. Information about the health RCS approaches, tensions and suggested solutions was re-constructed into a narrative. Throughout the process contacts in the health RCS funder agencies were involved in helping us to validate and interpret our results. The focus of the health RCS evaluations ranged from individuals and institutions to national, regional and global levels. Our analysis identified tensions around how much stakeholders should participate in an evaluation, the appropriate balance between measuring and learning and between a focus on short-term processes vs longer-term impact and sustainability. Suggested solutions to these tensions included early and ongoing stakeholder engagement in planning and evaluating health RCS, modelling of impact pathways and rapid assimilation of lessons learned for continuous improvement of decision making and programming. The use of developmental approaches could improve health RCS evaluations by addressing common tensions and promoting sustainability. Sharing learning about how to do robust and useful health RCS evaluations should happen alongside, not after, health RCS efforts.

Designing and measuring the progress and impact of health research capacity strengthening initiatives

“…Stakeholders such as funding recipients have greater in-depth knowledge about the project and its context compared to external evaluators, which is vital for solving problems and sustainability. However there is a tension between this approach, which takes time and needs an additional, ear-marked budget, and the traditional summative model of evaluation carried out by an independent external team, which many funders' perceive as better value for money, for demonstrating accountability and for producing rapid results[ 10 ].…”

Qualitative study to develop processes and tools for the assessment and tracking of African institutions’ capacity for operational health research

“…17 However, despite long-standing calls for more robust evaluations of capacity development, 11 the evidence needed to inform effective implementation and evaluation of programmes for strengthening operational health research capacity remains weak. 18 19 Furthermore, the lack of clearly defined goals and baselines against which to evaluate the success of research capacity strengthening programmes makes it difficult to track their progress and impact. 20 …”

Integrating capacity development during digital health research: a case study from global health

“…The recently published Essence Framework was developed by a group of funders whose secretariat is located at the Special Programme for Research and Training in Tropical Diseases (TDR) of the WHO, and is used as a framework to encourage best practices in research capacity strengthening globally [9]. It was designed to evaluate capacity development activities across health fields, and use cases are beginning to emerge in the literature [17][18][19]. The Essence Framework divides capacity building activities into four primary levels: individual, organizational, national and regional systems and research network.…”