Applying a systems and complexity lens to building evaluation capacity: Learning from a multicountry donor's experience

Vang, KaYing; Moore, Marah; Nicklin, Claire

doi:10.1002/ev.20463

Cited by 2 publications

(1 citation statement)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…There is an increasing amount of literature arguing for complexity-informed practice that places inordinate demands on the evaluator (Barnes et al, 2003;Sridharan & Nakaima, 2020;Vang et al, 2021). The range and skills required of evaluators to navigate the kind of complexity that is the norm today is ever expanding (Diaz et al, 2020).…”

Section: Discussionmentioning

confidence: 99%

Reflections on Required Competencies For Health Systems Monitoring, Evaluation, and Learning In India

Rao,

Nambiar

2023

J Multidiscip Eval

View full text Add to dashboard Cite

The movement towards systems thinking and complexity-informed monitoring and evaluation has been ongoing for some time. Covid-19 has accelerated this shift and increased the salience of contextually aware, adaptive forms of evaluation. Drawing from our experience over three years of providing monitoring, evaluation, research and learning (MERL) to a large international philanthropic organization’s health systems portfolio of work in India, we identify three types of evaluator capacities and associated set of competencies that derive from complexity-driven practice in a developing country. The first type of capacity relates to deriving evidence in contexts where there are very few traditional forms of data. The MERL portfolio related to health systems outcomes of access, financial protection, and equity even as the health programs themselves were evolving in a period of rapid transformation and dynamism of policy priorities and implementation mechanisms. This required an integration of traditional performance indicators drawn from government datasets with qualitative ‘signals’ focused on drivers of change within the system. Assessment of signals in turn required synthesizing different sources of information, including informal channels of obtaining evidence such as partner meetings or government-sponsored events. The evaluating team thus consciously empowered different kinds of researchers with differential expertise with the goal of building a much more pixelated picture. The goal was not to identify a single source of truth but rather a patchwork of validated information where the relevance of different pieces of data were dependent on evolving outcomes of interest. The second set of competencies related to the skills required to play a convening role for donors and implementing partners, supporting better understanding of the changing operating context and help inform decision-making by program officers and partners. This involved building and sustaining relationships across different stakeholders at different stages of the project – from proposal development to review. Competencies relating to effective dialogue and developing an understanding of the core interests of international, national and sub-national partners as well as international donors and experts could only be developed iteratively and over time, but this was crucial in a distributed health decision-making ecosystem like India. The third and final set of competencies relate to operational adaptiveness, while evaluating an ecosystem with few constants. This can be the hardest competency to acquire because it is the farthest from the traditional notions embedded in the training of evaluation scientists and MERL practitioners. We found that the degree of buy-in and commitment to previously agreed upon frames of reference for evaluation can be shifted by changes in personnel or internal organizational structures. These shifts can lead to chain reactions of mismatched expectations that needed to be understood and managed in real time by MERL partners. The pandemic further created a natural experiment that on the one hand required a reexamination of program priorities and on the other depended on reliability of donor support. Each of these three types of capacities – synthesizing nuanced evidence for adaptive action, relationship building and communication, and managing operational discontinuities are in fact inter-dependent. Building evaluator competencies isn’t simply about capacity-building but rather a recognition of the diversity of skills and worldviews that need to be encompassed within our monitoring and evaluation functions for today’s complex, discontinuous health systems.

show abstract

Section: Discussionmentioning

confidence: 99%

Reflections on Required Competencies For Health Systems Monitoring, Evaluation, and Learning In India

Rao,

Nambiar

2023

J Multidiscip Eval

View full text Add to dashboard Cite

show abstract

Exploring the Edges: Identifying the Next Generation of Evaluation Capacity Building Research and Practice Through Adjacency

Fierro,

Bourgeois

2024

Canadian Journal of Program Evaluation

View full text Add to dashboard Cite

Evaluation capacity building (ECB) is a well-established domain of scholarship and practice within the broader field of evaluation. Stemming from a concern to ensure sustainable and useful evaluation practices in organizations, ECB draws on several evaluation theories and approaches such as collaborative evaluation and utilization-focused evaluation. In a recent integrative review, the authors and their colleagues identified other theories, frameworks, and models that inform ECB; many of these were developed in other fields, such as social sciences and public administration. The present exploratory study sought to identify some of the knowledge and theories that inform ECB from other fields through a review of 47 “ECB-adjacent” articles. This article describes the various themes and topics included in these articles to formulate potential future avenues for ECB research and practice.

show abstract

Applying a systems and complexity lens to building evaluation capacity: Learning from a multicountry donor's experience

Cited by 2 publications

References 21 publications

Reflections on Required Competencies For Health Systems Monitoring, Evaluation, and Learning In India

Reflections on Required Competencies For Health Systems Monitoring, Evaluation, and Learning In India

Exploring the Edges: Identifying the Next Generation of Evaluation Capacity Building Research and Practice Through Adjacency

Contact Info

Product

Resources

About