Learning from seasoned evaluators: Implications of systems approaches for evaluation practice

Abstract

In the last 20 years, approaches (i.e. concepts, theories, methods) from the systems fields have increased in popularity among evaluators and evaluation commissioners. Their popularity stems in part from a widespread view that systems approaches transform evaluation practice, despite a lack of evidence. There have been few empirical examinations of the implications of these approaches for evaluation practice. This study analyzed eight cases of seasoned evaluators using systems approaches in diverse areas of evaluation practice to identify implications for how these professionals practice evaluation. Case materials included semi-structured interviews with each evaluator and descriptions of evaluations they conducted. Findings support a variety of case-specific transformations rather than a wholesale transformation in evaluation practice. Future directions for evaluator training, evaluation commissioning and contracting, and research are identified.

How do policy evaluators understand complexity?

“…They have considered, for example, how complexity may provide a mandate for going beyond limited or narrow theories of change (Garcia and Zazueta, 2015), how evaluators can develop complexity-consistent theory in relation to realist evaluation (Westhorp, 2012(Westhorp, , 2013, how they can develop complexity-appropriate evaluation questions (Larson, 2018), or match approaches to different levels of complexity and non-linearity (Ofek, 2016) or interventions in real-time (Ling, 2012). Gates (2017) also directly analyses how, in eight specific examples, systems approaches were used by experienced evaluators.…”

How does the commissioning process hinder the uptake of complexity-appropriate evaluation?

“…The differences between commissioners and evaluators is a common theme; Schneider et al (2016) and Broer et al (2017) find that evaluators and commissioners sometimes differ in what they perceive as 'good' evaluation and typically have different views and priorities. Gates (2017) suggests systems and complexity approaches can be a way to bridge this gap and be used to reconfigure the relationship between commissioners and evaluators, away from a client-service-type relationship, towards partnership models involving shared question setting, negotiated and flexible contracts, and a high level of involvement of commissioners in the actual evaluation activities.…”

Policy evaluation for a complex world: Practical methods and reflections from the UK Centre for the Evaluation of Complexity across the Nexus

“…Articles have considered how complexity can inform existing approaches (e.g. Garcia and Zazueta, 2015; Gates, 2017; Reynolds, 2015). Others have thought about how to develop complexity-appropriate evaluation questions (e.g.…”

“…Callaghan, 2008) and then more recently beginning to review applications (e.g. Gates, 2016Gates, , 2017Mowles, 2014;Ofek, 2016;Reynolds et al, 2016;Walton, 2014Walton, , 2016aWilliams, 2015). There is also a strong 'realist' bent to the complexity work during these years, with eight articles in this journal having the term 'realist' or 'realism' in the title (e.g.…”

“…Larson, 2018). Gates (2017) and Barbrook-Johnson et al (2020) also directly consider how systems and complexity approaches are understood and used by evaluators.…”